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Executive  Summary 


Motivation 

This  paper  examines  whether  Major  Defense  Acquisition  Programs  (MDAPs)  that 
entered  a  boom  climate  for  procurement  funding  some  time  after  passing  Milestone  (MS) 
B  on  average  had  higher  unit  cost  growth  than  programs  whose  acquisition  cycles  did  not 
extend  into  a  boom  climate.  While  this  conjecture  seems  plausible,  possibly  even 
obviously  correct,  it  has  not  been  recognized  in  the  cost  growth  literature. 

The  topic  is  worth  pursuing  because  it  bears  on  why  unit  cost  growth  was 
significantly  higher  for  MDAPs  that  passed  MS  B  in  bust  periods  than  it  was  for  those 
that  passed  in  boom  periods.  This  observation  was  reported  by  the  first  paper  in  this 
series,  hereafter  P-5126.1  The  explanation  offered  there  was  a  version  of  the  “camel’s 
nose”  hypothesis — that  unrealistic  cost,  programmatic,  and  technological  assumptions  are 
made  in  the  hope  that,  by  making  the  program  appear  to  be  lower  in  cost  or  more  capable, 
they  will  increase  the  odds  that  the  program  will  be  successful  in  competition  for  funds. 
P-5126  goes  further  by  suggesting  that  the  incentives  for  adopting  very  optimistic 
assumptions  are  stronger  for  programs  that  pass  MS  B  in  bust  funding  climates  than  they 
are  for  programs  that  pass  in  boom  periods,  and  that,  consequently,  unrealistic  MS  B 
baselines  are  more  common  in  programs  that  passed  MS  B  in  bust  climates. 

For  present  purposes,  the  key  point  to  note  is  that  the  explanation  offered  by  P-5 126 
supposes  that  most  of  the  growth  in  unit  cost  shown  by  programs  that  pass  MS  B  in  a  bust 
funding  climate  is  “baked  into”  the  baselines  established  at  MS  B.  There  are  other 
possibilities,  however,  one  of  which  is  that  a  significant  part  of  these  programs’  cost 
growth  might  be  due  to  increases  in  program  content  made  during  a  post-MS  B  boom 
climate  when  funding  is  more  readily  available.  To  the  extent  that  is  the  case,  we  may  be 
mistaking  the  costs  of  decisions  to  improve  the  capabilities  of  an  existing  system  for 
growth  in  the  costs  of  acquiring  the  capabilities  specified  in  the  MS  B  baseline. 

Approach 

The  most  analytically  satisfactory  approach  to  exploring  the  existence  and  character 
of  boom  effects  would  be  to  look  in  detail  at  content  changes  for  programs  that  did  and 
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did  not  experience  a  boom  funding  climate  after  passing  MS  B.  That  would  require  far 
more  resources  than  were  available  for  this  paper,  however.  Instead,  this  paper  uses  a 
statistical  approach  that  relies  on  data  that  are  comparatively  easy  to  acquire  and  natural 
experiments  present  in  the  historical  record. 

In  the  language  of  medical  testing,  the  plan  is  to  compare  unit  cost  growth  for  a 
treatment  group — programs  that  experienced  a  boom  climate  post  MS  B — with  that  of  a 
control  group — programs  that  did  not.  The  measure  of  cost  used  is  Program  Acquisition 
Unit  Cost  (PAUC).  PAUC  growth  is  computed  by  comparing  the  MS  B  baseline  value 
for  PAUC — which  can  be  thought  of  as  a  goal  or  a  prediction — to  the  actual  PAUC 
reported  in  the  final  Selected  Acquisition  Report  (SAR)  for  the  program,  normalized  to 
the  MS  B  quantity.  Both  the  MS  B  value  and  the  actual  value  of  PAUC  are  stated  in 
constant  dollars.2  In  what  follows,  PAUC  growth  means  PAUC  growth  normalized  to  the 
MS  B  quantity.  Note  that  we  do  not  nonnalize  PAUC  growth  for  any  changes  in  the 
capabilities  that  the  program  acquires. 

We  first  compare  the  average  of  PAUC  growth  for  MDAPs  that  passed  MS  B  in 
bust  climates  and  did  not  continue  into  a  subsequent  boom  with  the  average  for  MDAPs 
that  passed  MS  B  in  a  bust  climate  and  went  on  into  a  boom  climate.  The  control  group  in 
this  scenario  is  programs  that  passed  MS  B  in  a  bust  period  but  were  completed  before 
the  subsequent  boom  period.  A  second  natural  experiment  is  provided  by  programs  that 
passed  MS  B  during  the  boom  of  the  early  to  mid-1980s  and  were  still  underway  in  the 
post- 9/1 1  boom. 

The  analysis  takes  account  of  two  other  factors  that  may  systematically  influence 
average  PAUC  growth  over  a  period  of  years.  The  first  of  these  is  acquisition  policy  and 
process.  The  paper  employs  three  acquisition  policy  and  process  bins: 

•  McNamara-Clifford  (Fiscal  Year  (FY)  1964-FY  1969) 

•  Defense  Systems  Acquisition  Review  Council/Defense  Acquisition  Board 
(DS ARC/DAB,  FY  1970-FY  1993  and  FY  2003-FY  2007) 

•  Acquisition  Reform  (AR,  FY  1994-FY  2002) 

McNamara-Clifford  and  AR  are  statistically  similar  to  each  other  but  statistically 
different  from  DSARC/DAB.3  The  text  of  the  paper  considers  the  DSARC/DAB  period, 
which  includes  two  separate  bust  periods  and  two  boom  periods.  Appendix  C  provides 
the  sparse  results  obtained  for  McNamara-Clifford  and  AR. 
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See  Appendix  B;  pages  B-5  through  B-9  explain  the  PAUC  growth  estimates  used  in  this  paper. 

See  David  L.  McNicol  et  at,  “Further  Evidence  on  the  Effect  of  Acquisition  Policy  on  Cost  Growth  of 
Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5330  (Revised)  (Alexandria,  VA:  Institute  for 
Defense  Analyses,  August  2016). 
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The  second  factor  considered  is  program  duration.  Not  surprisingly,  programs  that 
fall  into  the  treatment  groups — i.e.,  those  that  passed  into  a  boom  period  post  MS  B — had 
considerably  longer  durations  on  average  than  did  programs  in  the  control  groups. 
Consequently,  we  need  to  examine  the  extent  to  which  the  longer  average  duration  of 
programs,  as  well  as  an  encounter  with  a  boom  funding  climate,  accounts  for  their  higher 
PAUC  growth.  This  was  done  for  programs  that  passed  MS  B  in  bust  periods  by 
estimating  a  model  that  includes  two  explanatory  variables:  (1)  the  number  of  years  that 
the  program  spent  in  boom  climates  post  MS  B;  and  (2)  the  number  of  years  the  program 
spent  in  bust  climates.  A  similar  model  was  estimated  for  programs  that  passed  MS  B 
during  one  of  the  boom  periods.  This  paper  finds  that  for  MDAPs  that  passed  MS  B  in 
bust  climates,  PAUC  increases  by  about  1.6  percent  for  each  year  the  program  spends  in 
a  bust  climate  and  by  about  5  percent  for  each  year  spent  in  a  boom  climate  post  MS  B. 

Conclusions  and  Limitations 

This  paper,  like  earlier  papers  in  the  series,  observes  that  PAUC  growth  measured 
from  MS  B  is  significantly  higher  in  programs  that  passed  MS  B  in  bust  climates  than  in 
boom  climates.  Moreover,  among  MDAPs  that  passed  MS  B  in  a  bust  phase  of 
DSARC/DAB,  only  those  that  continued  into  a  boom  climate  showed  PAUC  growth 
significantly  higher  than  that  of  programs  that  passed  MS  B  in  a  boom  climate.  This 
conclusion  is  important  because  it  implies  that  much  of  the  observed  PAUC  growth  may 
have  causes  other  than  flaws  in  MS  B  baselines. 

The  conclusion  tells  us  less  than  might  be  hoped,  however.  This  is  so  because  the 
PAUC  growth  associated  with  the  boom  may  reflect  purchase  of  capability  beyond  that 
specified  in  the  MS  B  baseline  or,  alternatively,  PAUC  increases  that  occur  when 
programs  take  advantage  of  a  boom  climate  to  “get  well.” 

The  average  PAUC  growth  of  all  programs  in  DSARC/DAB  that  passed  MS  B  in 
bust  climates  is  38  percent.  Without  making  a  specific  estimate,  P-5126  suggested  that 
most  of  this  PAUC  growth  stemmed  from  flawed  MS  B  baselines.  If  all  of  the  unit  cost 
growth  is  actually  a  matter  of  “getting  well,”  the  PAUC  growth  due  to  flawed  MS  B 
baseline  problems  remains  at  38  percent.  It  is  less  than  38  percent  to  the  extent  that 
PAUC  growth  for  MDAPs  that  passed  MS  B  in  bust  periods  and  continued  into  a  boom 
period  acquires  capabilities  beyond  these  of  the  MS  B  baselines.  Similarly  cost  accretion 
during  years  spent  in  bust  climates  that  does  acquire  capability  beyond  that  of  the  MS  B 
baseline  should  be  deducted. 

A  substantial  part  of  the  PAUC  growth  observed  in  MDAPs  that  continued  into  a 
boom  climate  post  MS  B  probably  does  reflect  acquisition  of  capabilities  beyond  that  of 
the  MS  B  baseline.  Unfortunately,  we  do  not  have  a  marker  for  PAUC  growth  due  to 
acquisition  of  additional  capability  (as  opposed  to  that  due  to  recognition  that  the  costs  of 
the  MS  B  capability  are  higher  than  anticipated  at  MS  B).  Further  statistical  analysis 
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along  the  lines  of  that  presented  here  seems  unlikely  to  be  useful  in  untangling  these  two 
elements.  Instead,  further  insight  into  why  programs  that  continued  into  a  boom  climate 
post  MS  B  had  higher  PAUC  growth  probably  will  require  detailed  examination  of 
changes  in  the  relevant  programs  post  MS  B. 
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A.  Introduction 


This  paper  examines  whether  Major  Defense  Acquisition  Programs  (MDAPs)  that 
entered  a  boom  climate  for  procurement  funding  some  time  after  passing  Milestone  (MS) 
B  on  average  had  higher  unit  cost  growth  than  programs  whose  acquisition  cycles  did  not 
extend  into  a  boom  climate.  While  this  conjecture  seems  plausible,  possibly  even 
obviously  correct,  it  has  not  been  recognized  in  the  cost  growth  literature. 

The  topic  is  worth  pursuing  because  it  bears  on  why  unit  cost  growth  was 
significantly  higher  for  MDAPs  that  passed  MS  B  in  bust  periods  than  it  was  for  those 
that  passed  in  boom  periods.  This  observation  was  reported  by  the  first  paper  in  this 
series,  hereafter  P-5126.1  The  explanation  offered  there  was  a  version  of  the  “camel’s 
nose”  hypothesis — that  unrealistic  cost,  programmatic,  and  technological  assumptions  are 
made  in  the  hope  that,  by  making  the  program  appear  to  be  lower  in  cost  or  more  capable, 
they  will  increase  the  odds  that  the  program  will  be  successful  in  competition  for  funds. 
P-5126  goes  further  by  suggesting  that  the  incentives  for  adopting  very  optimistic 
assumptions  are  stronger  for  programs  that  pass  MS  B  in  bust  funding  climates  than  they 
are  for  programs  that  pass  in  boom  periods,  and  that,  consequently,  unrealistic  MS  B 
baselines  are  more  common  in  programs  that  passed  MS  B  in  bust  climates. 

For  present  purposes,  the  key  point  to  note  is  that  the  explanation  offered  by  P-5 126 
supposes  that  most  of  the  growth  in  unit  cost  shown  by  programs  that  pass  MS  B  in  a  bust 
funding  climate  is  “baked  into”  the  baselines  established  at  MS  B.  There  are  other 
possibilities,  however,  one  of  which  is  that  a  significant  part  of  these  programs’  cost 
growth  might  be  due  to  increases  in  program  content  made  during  a  post-MS  B  boom 
climate  when  funding  is  more  readily  available.  To  the  extent  that  is  the  case,  we  may  be 
mistaking  the  costs  of  decisions  to  improve  the  capabilities  of  an  existing  system  for 
growth  in  the  costs  of  acquiring  the  capabilities  specified  in  the  MS  B  baseline. 

B.  Framework 

The  topic  of  this  paper  requires  distinguishing  between  bust  funding  and  boom 
funding  climates.  The  period  Fiscal  Year  (FY)  1965-FY  2009  considered  here  spans  two 
bust-boom  cycles  in  Department  of  Defense  (DoD)  procurement  funding:  (1)  The  bust 
climate  for  modernization  of  weapon  systems  that  began  in  the  mid-1960s2  and  lasted 
until  the  Carter-Reagan  buildup  of  the  early  to  mid-1980s,  and  (2)  the  long  post-Cold 


1  David  L.  McNicol  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy  and  Process  on 
Cost  Growth  of  Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5126  (Alexandria,  VA:  Institute 
for  Defense  Analyses,  September  2014),  5. 

2 

"  The  DoD  budget  was  high  during  the  years  of  the  Vietnam  War,  but  much  of  the  acquisition  budget 
went  for  munitions  and  to  weapon  systems  lost  in  combat.  Consequently,  funding  for  major  system  new 
starts  was  relatively  constrained.  See  Appendix  A,  page  A-8. 
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War  bust  climate  followed  by  the  post-9/11  boom.  The  rationales  for  the  break  points 
between  the  funding  climates  are  provided  in  Section  C  of  Appendix  A. 

A  measure  of  cost  growth  also  is  required.  One  option  is  based  on  Program 
Acquisition  Unit  Cost  (PAUC).  PAUC  is  the  sum  of  Research,  Development,  Test  and 
Evaluation  (RDT&E)  cost  and  procurement  cost,  divided  by  the  number  of  units 
acquired.  For  this  study,  PAUC  growth  is  computed  by  comparing  the  MS  B  baseline 
value  of  PAUC — which  can  be  thought  of  as  a  goal  or  a  prediction — to  the  actual  PAUC 
reported  in  the  last  Selected  Acquisition  Report  (SAR)  for  the  program,  normalized  to  the 
MS  B  quantity.3  (The  quantity  normalization  computations  are  described  in  Appendix  B, 
pages  B-7  to  B-9.)  Both  the  MS  B  baseline  and  the  actual  value  of  PAUC  are  stated  in 
constant  dollars.  The  alternative  to  PAUC  growth  is  growth  in  Average  Procurement  Unit 
Cost  (APUC),  which  does  not  include  RDT&E  cost.  The  effects  of  changes  in  the 
capabilities  procured  may  be  more  likely  to  show  up  clearly  in  APUC  growth,  which  is 
an  advantage,  but  it  is  a  less  comprehensive  measure  of  unit  cost  growth.  We  compute  the 
results  for  both  cost  growth  measures,  and  report  the  results  for  APUC  only  in  the  one 
instance  in  which  when  they  differ  in  an  important  way  from  those  obtained  using  PAUC. 
Note  that  PAUC  growth  and  APUC  growth  are  adjusted  for  quantity  but  not  for  changes 
in  the  capabilities  the  program  is  directed  to  acquire. 

In  what  follows,  the  term  PAUC  growth  means  PAUC  growth  from  the  MS  B 
Baseline  with  the  final  SAR  PAUC  normalized  to  the  MS  B  quantity.  Similarly,  the  term 
APUC  growth  means  APUC  growth  from  the  MS  B  Baseline  with  the  final  SAR  APUC 
normalized  to  the  MS  B  quantity.  Appendix  B  provides  the  conventions  used  in 
assembling  the  database  and  the  sources  of  the  data  used.  The  unit  cost  growth  estimates 
were  updated  to  the  most  recent  comprehensive  infonnation  available,  that  in  the 
December  2015  SARs.  Only  completed  programs  (defined  as  programs  with  an  end  date 
of  FY  2016  or  earlier)  are  used  in  this  analysis  because  some  costs  associated  with  a 
program  may  not  be  fully  reflected  in  its  SAR  until  the  program  is  completed. 

Average  PAUC  growth  reported  in  Table  1  for  programs  that  passed  MS  B  in  bust 
climates  is  significantly  higher  (43  percent)  than  it  is  for  programs  that  passed  MS  B  in 
boom  periods  (15  percent).4  This  observation  serves  only  to  confirm  for  the  data  used  in 
this  research  the  result  mentioned  above  from  P-5 126. 


3 

PAUC  and  APUC  growth  measures  used  for  purposes  of  Nunn-McCurdy  Act  reporting  are  not  quantity 
normalized.  The  median  MDAP  that  passed  MS  B  in  the  period  FY  1988-FY  2007  acquired  100 
percent  of  MS  B  baseline  quantity,  and  the  average  program  acquired  111  percent.  See  David  L. 
McNicol,  Sarah  K.  Burns,  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy  and 
Process  and  Funding  Climate  on  Cancellation  of  Major  Defense  Acquisition  Programs”  IDA  Paper 
P-5218  (Alexandria,  VA:  Institute  for  Defense  Analyses,  May  2015),  7-8.  Compared  to  the  PAUC 
growth  measures  used  in  Nunn-McCurdy  reporting,  quantity  adjusted  decreased  measured  PAUC 
growth  for  programs  for  about  half  of  the  programs  in  the  sample  and  increased  it  for  the  other  half. 

4  P  <  0.001  for  the  Mann-Whitney  U  (M-W  U)  test  (U  =  1261.5,  m  =  108,  n2  =  44). 
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Table  1.  Average  PAUC  Growth  for  Completed  MDAPs  by  MS  B  Funding  Climate 


Bust 

Boom 

FY  1964-FY  1980 

47%  (64) 

FY  1981-FY  1986 

18%  (35) 

FY  1987-FY  2002 

37%  (44) 

FY  2003-FY  2009 

0%  (9) 

Total 

43%  (108) 

Total 

15%  (44) 

Note:  Numbers  of  MDAPs  that  passed  MS  B  and  were  completed  by  the  December  2015  SARs  are  shown 
in  parentheses. 


Finally,  it  is  necessary  to  recognize  changes  over  time  in  acquisition  policy  and 
process  configurations  because  they  are  associated  with  significant  difference  in  average 
PAUC  growth.  A  previous  paper  in  this  series  (P-5330  (Revised)5)  distinguished  the 
following  six  policy  and  process  configurations: 

1.  McNamara-Clifford  (FY  1964-FY  1969) 

2.  Defense  Systems  Acquisition  Review  Council  (DSARC,  FY  1970-FY  1982) 

3.  Post-Carlucci  DSARC  (P-C  DSARC,  FY  1983-FY  1989) 

4.  Defense  Acquisition  Board  (DAB,  FY  1990-FY  1993) 

5.  Acquisition  Refonn  (AR,  FY  1994-FY  2000) 

6.  DAB  Post  AR  (DAB  Post  AR,  FY  200 1-FY  2009) 

Average  PAUC  growth  does  not  differ  significantly  among  DSARC,  P-C  DSARC,  DAB, 
and  DAB  Post  AR  within  a  budget  climate.6  Their  statistical  similarity  pennits  these 
periods  to  be  combined  into  a  single  acquisition  policy  and  process  configuration,  which 
will  be  referred  to  as  DS ARC/DAB.  The  main  text  is  concerned  only  with  the 
DS ARC/DAB.  A  previous  paper  in  this  series  found  that  average  APUC  growth  was 
significantly  higher  in  McNamara-Clifford  and  AR  configurations,  which  for  that  reason 


5  David  L.  McNicol  et  al.,  “Further  Evidence  on  the  Effect  of  Acquisition  Policy  on  Cost  Growth  of 
Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5330  (Revised)  (Alexandria,  VA:  Institute  for 
Defense  Analyses,  August  2016). 

6  For  bust  climates,  Analysis  of  Variance  (ANOVA)  fails  to  reject  the  null  hypothesis  that  APUC  growth 
for  completed  programs  in  each  of  these  bins  has  the  same  normal  distribution  (P  =  0.996). 
Kolmogorov-Smirnov  (K-S),  Anderson  Darling  (A-D),  and  an  F-test  of  the  variances  indicate  that  the 
assumptions  of  ANOVA  are  satisfied.  For  boom  climates,  K-S  and  A-D  find  that  the  observations  for 
the  boom  portion  of  DSARC  and  the  DAB  periods  are  consistent  with  a  normal  distribution,  but  K-S 
rejects  normality  for  the  boom  portion  of  P-C  DSARC.  The  M-W  U  test  does  not  detect  a  significant 
difference  between  the  means  of  the  (1)  DSARC-Boom  and  P-C  DSARC-Boom  (P  =  0.968,  U  =  88.5, 
ni  =  29,  n2  =  6);  (2)  DSARC-Boom  and  DAB  Post  AR-Boom  (P  =  0.317,  U  =  36,  m  =  9,  n?  =  6);  or  (3) 
the  P-C  DSARC-Boom  and  the  DAB  Post  AR-Boom  (P  =  0.215,  U  =  94,  m  =  29,  n2  =  9). 
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are  treated  separately.7  Appendix  C  provides  results  for  the  McNamara-Clifford  and  AR 
configurations. 

Section  B  of  Appendix  A  provides  brief  descriptions  of  the  acquisitions 
configurations  as  defined  here.  Readers  who  have  read  previous  papers  in  this  series  or 
who  are  generally  familiar  with  the  Office  of  the  Secretary  of  Defense  (OSD)-level 
acquisition  process  and  various  acquisition  reform  efforts  can  use  Appendix  A 
selectively.  Others  may  wish  to  read  Appendix  A  before  proceeding  with  the  main  text. 

C.  Evidence  of  a  Boom  Effect 

The  term  boom  effect  is  used  here  to  label  a  feature  observed  in  the  unit  cost  growth 
data — MDAPs  that  passed  through  a  boom  climate  post  MS  B  had  a  higher  average  unit 
cost  growth  than  those  that  did  not. 

Many  MDAPs  that  passed  MS  B  in  one  of  the  bust  climates  continued  into  a  boom 
climate,  and  some  programs  that  passed  MS  B  during  the  Carter-Reagan  defense  buildup 
continued  into  the  post- 9/ 11  boom.  A  two-part  naming  convention  is  used  to  label  two 
bins  of  programs:  those  that  did,  and  those  that  did  not,  pass  through  a  boom  climate  post 
MS  B.  The  first  part  of  the  label  gives  the  funding  climate  prevailing  when  the  program 
passed  MS  B — bust  or  boom.  The  second  part — 0,  1,  or  2 — denotes  the  number  of  boom 
climates  a  program  passed  through  post  MS  B.  For  example,  programs  that  were 
completed  entirely  within  a  single  bust  phase  will  be  referred  to  as  BustO — Bust  because 
they  passed  MS  B  in  a  bust  funding  climate  and  zero  because  they  were  completed 
without  entering  a  boom  climate.  Programs  that  passed  MS  B  in  a  bust  period  and 
continued  into  or  through  a  subsequent  boom  period  are  called  Bustl. 

A  detailed  evaluation  of  content  changes  for  programs  that  did  and  did  not 
experience  a  boom  funding  climate  after  passing  MS  B  would  be  the  best  approach  to 
exploring  the  importance  and  character  of  boom.  This  type  of  analysis  would  require 
greater  resources  than  were  available,  however.  Instead,  this  paper  uses  a  statistical 
approach  that  relies  on  data  that  are  comparatively  easy  to  acquire — PAUC  growth  from 
the  MS  B  baseline,  the  year  programs  passed  MS  B,  and  the  year  the  programs  were 
completed.  In  the  language  of  medical  testing,  the  plan  is  to  compare  unit  cost  growth  for 
a  treatment  group — programs  that  experienced  a  boom  climate  post  MS  B — with  that  of  a 
control  group — programs  that  did  not.  The  question  asked  in  this  section  is  whether  the 
observed  boom  effects  are  statistically  significant.  We  look  first  at  the  two  bust  climates 
and  then  at  the  two  boom  climates. 


7  McNicol  et  at,  “Further  Evidence  on  the  Effect  of  Acquisition  Policy,”  Appendix  C,  which  provides  a 
Bayesian  analysis  using  APUC  growth  data.  That  result  also  probably  holds  for  the  PAUC  data. 
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PAUC  growth  for  BustO  and  Bustl  are  presented  in  Table  2  for  each  of  the  two  bust 
periods  of  DS ARC/DAB.  In  both  periods,  average  PAUC  growth  for  the  treatment  group 
(Bustl)  is  higher  than  it  is  for  the  control  group  (BustO) — 42  percent  compared  to  16 
percent  for  the  first  period  and  51  percent  compared  to  13  percent  for  the  second.  These 
differences  are  statistically  significant. 8  For  programs  that  passed  MS  B  in  a  bust  period, 
subsequent  entry  into  a  boom  period  is  then  associated  with  higher  PAUC  growth. 

Table  2.  Average  PAUC  Growth  for  Completed  MDAPs  in  DSARC/DAB  Bust  by  the  Number 

of  Boom  Periods  Experienced 


Bin 

1st  Bust  Period 

FY  1970-FY  1980 

2nd  Bust  Period 

FY  1987-FY 1993 

BustO 

16%  (6) 

13%  (8) 

Bustl 

43%  (39) 

51%  (17) 

Bust2 

19%  (3) 

none 

Bust2  does  not  follow  this  pattern:  average  PAUC  growth  for  Bust2  is  slightly 
higher  than  that  of  BustO  but  less  than  that  of  Bustl.  The  number  of  programs  in  this  bin 
(N=3),  however,  is  so  small  that  there  is  no  point  in  speculating  about  why  it  does  not  fit 
the  pattern.9  While  no  attempt  is  made  to  explain  the  observation  for  Bust2,  it  is  included 
in  an  analysis  discussed  below  that  includes  all  of  the  MDAPs  that  passed  MS  B  during 
the  two  DSARC/DAB  bust  periods. 

APUC  growth  also  does  not  entirely  follow  the  pattern  of  PAUC  growth  for  BustO 
and  Bustl  of  the  two  DSARC/DAB  bust  periods.  In  particular,  APUC  growth  for 
programs  initiated  in  the  first  DSARC/DAB  bust  period  does  not  show  a  statistically 
significant  boom  effect  in  APUC  growth.  (The  six  programs  of  BustO  have  an  average 
APUC  growth  of  21  percent,  which  is  not  significantly  different  from  the  42  percent 
average  APUC  for  the  39  programs  of  Bustl.10)  APUC  growth  in  the  second  bust  period 


K-S  and  A-D  find  the  PAUC  growth  data  in  each  of  the  two  bins  of  the  first  bust  period  to  be  consistent 
with  a  normal  distribution.  An  F-test  found  the  two  variances  to  be  significantly  different.  A  two-tailed 
t-test  assuming  unequal  sample  variances  found  the  means  of  Bustl  and  BustO  for  the  first  period  to  be 
significantly  different  (P  =  0.01 1).  K-S  and  A-D  also  find  the  PAUC  growth  data  in  each  of  the  two  bins 
of  the  first  bust  period  to  be  consistent  with  a  normal  distribution.  Again,  an  F-test  found  the  two 
variances  to  be  significantly  different.  A  two-tailed  t-test  assuming  unequal  sample  variances  found  the 
means  of  Bustl  and  BustO  for  the  first  period  to  be  significantly  different  (P  =  0.004). 

9  The  programs  in  Bust2  are  the  CNV  68,  with  a  PAUC  growth  of  7  percent;  the  NAVSTAR  GPS  (85 
percent);  and  ATCCS-MCS  (-34  percent). 

10  K-S  and  A-D  find  the  APUC  growth  data  in  each  of  the  two  bins  of  the  first  bust  period  to  be  consistent 
with  a  normal  distribution.  An  F-test  found  no  significant  difference  between  the  two  variances.  A  two- 
tailed  t-test  of  the  APUC  data  found  the  means  of  Bustl  and  BustO  for  the  first  period  not  to  be 
significantly  different  (P  =  0.241). 
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does  follow  the  pattern — 43  percent  for  Bustl,  which  is  significantly  higher  than  the  17 
percent  average  APUC  growth  for  BustO. 1 1 

Table  3  presents  data  on  PAUC  growth  for  the  two  DSARC/DAB  boom  periods. 
The  nomenclature  used  for  the  boom  periods  parallels  that  used  for  bust  periods.  BoomO 
programs  passed  MS  B  in  a  boom  climate  and  were  completed  in  that  boom  or  the 
succeeding  bust  climate.  Booml  programs  passed  MS  B  during  the  Carter-Reagan 
defense  buildup  and  were  completed  during  the  post-9/11  boom  or  during  the  following 
three  years.  There  is  no  treatment  group  (i.e.,  Booml)  for  the  second  boom  period  and 
hence  no  experiment  to  examine. 

Table  3.  Average  PAUC  Growth  for  Completed  MDAPs  in  DSARC/DAB-Boom  by  the 
Number  of  Boom  Periods  Experienced 


1st  Boom  Period 

2nd  Boom  Period 

Bin 

MS  B  FY  1981-FY  1986 

MS  B  FY  2003-FY  2009 

BoomO 

12%  (28) 

0%  (9) 

Booml 

45%  (7) 

none 

Average  PAUC  growth  for  the  Booml  programs  of  the  first  boom  period  (45 
percent)  is  significantly  higher  than  that  for  the  BoomO  programs  (12  percent). 12  This 
finding  is  somewhat  unexpected,  since  the  relevant  programs  passed  MS  B  in  a  boom 
funding  climate  and  presumably  had  realistic  baselines  and  were  robustly  funded  at  least 
initially.  In  fact,  the  finding  may  be  spurious.  Average  PAUC  growth  for  the  Booml  bin 
of  the  first  boom  period  is  dominated  by  three  MDAPs,  each  of  which  had  PAUC  growth 
of  more  than  40  percent:  C-17  (57  percent),  T-45  Goshawk  (70  percent),  and  JSTARS 
(123  percent).  These  programs  had  the  essential  features  of  Total  Package  Procurement 
(TPP).13  Acquisition  reforms  adopted  in  mid- 1969  ruled  out  use  of  TPP  and  fixed-price 
development  contracts  because  they  typically  resulted  in  severe  cost  growth  and  schedule 
problems.14  During  the  Reagan  Administration,  however,  TPP-like  contracts  were  used 


11  M-W  U  P  =  0.041  (U  =  32.5,  m  =  17,  n2  =  8). 

12 

~  K-S  found  the  distribution  of  APUC  growth  of  the  28  BoomO  programs  that  passed  MS  B  in  the  first 
bust  phase  to  be  non-normal.  M-W  U  found  the  difference  between  average  APUC  growth  of  BoomO 
and  Booml  for  the  first  boom  phase  to  be  significant.  (P  =  0.007,  U  =  164.5,  m  =  28,  n2  =  7). 

13 

David  L.  McNicol,  Cost  Growth  in  Major  Weapon  Procurement  Programs,  2nd  ed.  (Alexandria,  VA: 
Institute  for  Defense  Analyses,  2004),  62. 

14  See  McNicol  et  al.,  “Further  Evidence  on  the  Effect  of  Acquisition  Policy,”  7,  and  for  further  discussion 
of  TPP  and  FFP  development  contracts,  see  Karen  Tyson  et  al.,  “The  Effects  of  Management  Initiatives 
on  the  Costs  and  Schedules  of  Defense  Acquisition  Programs,  Vol.  I:  Main  Report,”  IDA  Paper  P-2722 
(Alexandria,  VA:  Institute  for  Defense  Analyses,  November  1992),  Chapter  X;  McNicol,  Cost  Growth, 
53  and  57-59;  and  William  D.  O’Neil  and  Gene  H.  Porter,  “What  to  Buy?  The  Role  of  Director  of 
Defense  Research  and  Engineering  (DDR&E) — Lessons  from  the  1970s,”  IDA  Paper  P-4675 
(Alexandria,  VA:  Institute  for  Defense  Analyses,  January  2011),  9-31. 
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for  a  few  MDAPs,  including  the  three  programs  noted  here.  (The  other  four  of  the  seven 
programs  in  Booml  had  conventional  cost  plus  incentive  fee  contracts  for  Engineering 
and  Manufacturing  Development  (EMD).)  The  PAUC  growth  of  the  C-17,  T-45,  and 
JSTARS  programs  was  on  a  par  with  that  of  TPP  programs  that  passed  MS  B  during 
FY  1965-FY  1969  and  did  not  continue  into  the  Carter-Reagan  boom.  Their  contracting 
strategy,  not  their  continuation  into  a  boom  funding  climate,  could  then  account  for  their 
high  PAUC  growth.  If  the  three  programs  are  excluded,  the  average  PAUC  growth  for 
Booml  is  17  percent,  which  is  not  significantly  higher  than  the  average  for  BoomO. 15 

Table  4  combines  data  from  Table  2  and  Table  3.  The  73  MDAPs  of  the 
DS ARC/DAB  bust  climates  had  an  average  PAUC  growth  of  38  percent,  which  was 
significantly  higher  than  the  9  percent  average  of  the  41  MDAPs  in  DSARC/DAB  that 
passed  during  boom  climates.16  Average  PAUC  growth  of  MDAPs  in  BustO  is  not 
significantly  different  from  the  average  PAUC  growth  of  DSARC/DAB  boom,  and 
therefore  has  little  effect  on  this  result.17  Instead,  the  higher  average  of  DSARC/DAB 
bust  is  mainly  due  to  the  programs  in  Bustl.  This  adds  an  important  point  to  the  narrative 
of  P-5 126:  the  higher  PAUC  growth  of  MDAPs  that  passed  MS  B  in  bust  climates  largely 
reflects  a  subset  associated  with  those  programs — those  that  passed  MS  B  in  a  bust 
climate  and  continued  on  into  a  boom  climate. 

Table  4.  PAUC  Growth  for  the  Combined  Bust  and  the  Combined  Boom  Phases  of 

DSARC/DAB 

BustO  14%  (14)  BoomO  9%  (37) 

Bustl  45%  (56)  Booml  17%  (4)* 

Combined  Bust**  38%  (73)  Combined  Boom  9%  (41) 

*  Excludes  C-17,  T-45,  and  JSTARS. 

**  Includes  the  three  programs  in  Bust2,  which  have  an  average  PAUC  growth  of  19 
percent. 

D.  Funding  Climate,  Program  Duration,  and  the  Boom  Effect 

This  section  takes  an  additional  step  towards  explaining  why  the  data  show  boom 
effects.  Table  5  presents  rearranged  data  from  Table  2  and  Table  3  and,  in  addition, 
shows  average  program  duration  for  each  bin.  Average  PAUC  growth  is  greater  in  Bustl 
than  in  BustO  for  each  of  the  two  bust  periods  and  greater  for  Booml  than  for  BoomO  for 
the  first  boom  period.  (The  second  boom  period  is  excluded  because  there  are  no 
programs  in  Booml.)  The  programs  in  Bustl  and  Booml,  however,  also  had  a  longer 


M-W  U  P  =  0.117  (UA=  83.5,  UB  =  28.5,  m  =  28,  n2  =  4). 
M-W  U  P  <  0.001  (U  =  633,  m  =  73,  n2  =  4). 

M-W  UP  =  0.121  (U  =  367.5,  m  =  41,  n2  =  14). 
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average  duration  than  the  programs  in  the  corresponding  BustO  and  BoomO  bins. 
Consequently,  we  need  to  examine  the  extent  to  which  longer  average  duration  in 
addition  to  an  encounter  with  a  boom  period  account  for  their  higher  PAUC  growth.  Note 
that  including  the  three  programs  of  Bust2  (of  the  first  bust  period)  and  the  three 
programs  excluded  from  Booml  would  not  change  this  conclusion. 

Table  5.  Average  PAUC  Growth  and  Average  Program  Duration  by  Number  of  Boom 
Periods  Encountered  for  Completed  Programs  in  DSARC/DAB 


Bin 

Average  PAUC 
Growth 

Average 
Duration  * 

1st  Bust  Period  * 

BustO 

16%  (6) 

6.7 

Bustl 

42%  (39) 

14.2 

2nd  Bust  Period 

BustO 

13%  (8) 

7.5 

Bustl 

53%  (17) 

15.8 

1st  Boom  Period 

BoomO 

12%  (28) 

9.2 

Boom1§ 

17%  (4) 

22.3 

+  From  MS  B  through  the  year  in  which  the  program’s  last  SAR  was  filed. 
*  Excludes  the  three  programs  of  Boom2. 

§  Excludes  C-17,  T-45,  and  JSTARS. 


We  approach  this  problem  by  dividing  the  duration  of  the  program  into  two  parts: 

1 .  Tboom  =  number  of  years  post  MS  B  spent  in  boom  climates 

2.  Tbust  =  number  of  years  post  MS  B  spent  in  bust  climates 

These  two  variables  are  hypothesized  to  have  distinct  linear  relationships  to  PAUC 
growth  (abbreviated  as  PAUC): 

PAUCi  ao  "F  aiTboomi  +  a2Tbusti  +  ei 

In  this  equation,  the  subscript  i  denotes  the  ith  MDAP  in  the  sample  and  ei  is  the  error 
tenn,  which  is  assumed  to  be  a  normally  distributed  random  variable.  The  coefficient  ai  is 
the  change  in  PAUC  for  each  year  the  program  spends  in  a  boom  climate.  Similarly,  a2  is 
the  change  in  PAUC  per  year  in  a  bust  climate.  The  estimated  intercept  tenn  ao  is  the 
average  net  effect  of  excluded  variables.  The  coefficients  of  the  model  are  estimated 
(using  multiple  regression)  separately  for  programs  that  passed  MS  B  in  bust  periods  of 
DSARC/DAB  and  those  that  passed  MS  B  in  its  first  boom  climate.  (The  second  boom 
climate  is  excluded  because  it  has  no  programs.)  The  estimates  obtained  are  presented  in 
Table  6. 18 


18 

An  alternative  to  the  model  above  posits  two  categories  of  MDAPs,  one  that  tends  to  short  duration  and 
low  unit  cost  growth  and  another  that  tends  to  long  duration  and  higher  unit  cost  growth.  Modifications 
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Table  6.  Years  in  Bust  Climates  and  Years  in  Boom  Climates  and  PAUC  Growth  for  MDAPs 
in  the  DSARC/DAB  Acquisition  Policy  and  Process  Configuration 

Passed  MS  B  in  Bust  Period* 

Passed  MS  B  in  Boom  Period* 

Estimate 

P-Value 

Estimate 

P-Value 

Intercept 

3.4% 

0.719 

3.7% 

0.608 

Years  in  Boom 

5.0%/yr“* 

<  0.001 

3.7%/yr“ 

0.039 

Years  in  Bust 

1 .6%/yr** 

0.042 

0.05%/yr 

0.937 

**  Statistically  significant  at  less  than  the  5  percent  level. 
“‘Statistically  significant  at  less  than  the  1  percent  level. 


*  R-Square  =  0.22  F  =  9.445  (P  <  0.001)  N=  70.  Estimated  by  Ordinary  Least  Squares  (OLS).  Excludes 
the  three  MDAPs  in  the  Bust2  bin  of  DSARC/DAB. 

*  R-Square  =  0.20  F  =  5.563  (P  =  0.002)  N=  32.  Estimated  by  OLS.  Excludes  C-17,  T-45,  and  JSTARS. 

Programs  that  passed  MS  B  in  a  bust  climate  characteristically  experienced  PAUC 
growth  of  1 .6  percent  for  each  year  spent  in  a  bust  climate.  PAUC  growth  for  each  year 
spent  in  a  boom  climate  post  MS  B  was  three  times  that  level — about  5  percent  per  year. 
Each  of  these  estimates  is  statistically  significant. 

The  effect  of  boom  years  for  programs  that  passed  MS  B  in  boom  periods  is  smaller 
(about  3.7  percent  per  year).  This  is  reasonable,  as  we  expect  programs  that  passed  MS  B 
in  boom  climates  to  have  realistic  baselines  and  to  be  adequately  funded  (at  least 
initially).  The  estimated  effect  per  bust  year  on  PAUC  growth  for  programs  that  passed 
MS  B  in  boom  periods  is  very  small  and  statistically  not  significant,  which  also  seems 
reasonable. 

A  sense  of  the  importance  of  the  boom  periods  entered  into  post  MS  B  is  provided 
by  Table  7.  The  table  shows  the  estimated  relationship  evaluated  at  the  sample  means  for 
T Boom  and  TBust  for  BustO  and  Bustl,  respectively.  Programs  in  BustO  have  an  average 
PAUC  growth  of  about  14  percent.  Of  this,  about  11.4  percentage  points  is  associated 
with  years  spent  in  bust  climates,  and,  of  course,  none  for  continuation  into  a  boom 
climate.  For  Bustl  programs,  boom  years  post  MS  B  account  for  about  26  percentage 
points  of  the  Bustl  average  PAUC  growth  of  45  percent;  the  years  spent  in  bust  climates 
account  for  15.2  percentage  points. 


and  upgrades  would  seem  to  be  examples  of  the  first  category  and  major  platforms  an  example  of  the 
second.  The  “short  duration"  and  “long  duration"  programs  were  defined,  respectively,  as  the  20 
percent  of  programs  in  the  bin  with  the  shortest  durations,  and  the  20  percent  with  the  longest  durations. 
The  results  for  all  of  the  forms  of  this  model  considered  rejecting  the  hypothesis  that  shorter  vice  longer 
duration  is  a  statistically  significant  factor  in  PAUC  growth. 
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Table  7.  Amount  PAUC  Growth  in  Boom  Climates  and  in  Bust  Climates  for  MDAPs  in 
DSARC/DAB  that  Passed  MS  B  in  Bust  Climates 


Period 

Intercept 

T  Boom 

TBust 

Average  PAUC  Growth 

BustO 

3.4% 

0 

1 1 .4% 

14.3%  (14) 

Bustl 

3.4% 

26% 

15.2% 

45%  (56) 

Note:  Evaluated  at  the  sample  means  for  TBoom  and  TBust. 


E.  Conclusions  and  Limitations 

This  paper,  like  earlier  papers  in  the  series,  finds  that  PAUC  growth  measured  from 
MS  B  is  significantly  higher  in  programs  that  passed  MS  B  in  bust  climates  than  in  boom 
climates.  Moreover,  among  MDAPs  that  passed  MS  B  in  a  bust  phase  of  DSARC/DAB, 
only  those  that  continued  into  a  boom  climate  showed  PAUC  growth  significantly  higher 
than  that  of  programs  that  passed  MS  B  in  a  boom  climate.  This  conclusion  is  important 
because  it  implies  that  much  of  the  observed  PAUC  growth  may  have  causes  other  than 
flaws  in  MS  B  baselines.  The  conclusion  tells  us  less  than  might  be  hoped,  however.  This 
is  so  because  the  PAUC  growth  associated  with  the  boom  may  reflect  the  purchase  of 
capability  beyond  that  specified  in  the  MS  B  baseline  or,  alternatively,  PAUC  increases 
that  occur  when  programs  take  advantage  of  a  boom  climate  to  “get  well.” 

The  Global  Broadcast  System  (GBS)  provides  an  example  of  a  program  whose 
content  was  increased  early  in  the  post-9/ 1 1  boom: 

The  current  GBS  architecture  is  based  on  Asynchronous  Transfer  Mode 
(ATM)  technology....  In  December  2002,  DoD  directed  GBS's  migration 
to  a  more  sustainable  commercial  and  standards-based  open  architecture, 
based  upon  the  Internet  Protocol  (IP).  Also,  the  GBS  program  received 
FY03  Iraqi  Freedom  Funds  (IFF)  supplemental  funding  for  IP 
Acceleration  of  production  units  to  replace  deployed  ATM  units.  Based 
upon  extensive  warfighter  inputs,  the  accelerated  IP  production  effort 
included  design  and  development  of  a  new,  single  case  version  of  the 
Receive  Suite  (88XR)  for  the  Army,  Navy,  and  Marine  Corps. 19 

Space  Based  Infrared  Satellite-High  (SBIRS-High)  is  a  convenient  and  useful  contrast  to 
GBS,  even  though  it  passed  MS  B  in  1997  and  hence  is  not  included  in  DSARC/DAB.  As 
of  the  December  2015  SARs,  funding  for  the  Baseline  SBIRS-High  program  was 
expected  to  end  in  FY  2018.  A  large  portion  of  the  growth  in  SBIRS-High  unit 
procurement  cost  for  the  baseline  program — roughly  one-third — occurred  before 
FY  2003,  while  most  of  the  other  two-thirds  occurred  during  FY  2003-FY  2009.  This 


19  Selected  Acquisition  Report:  Global  Broadcast  System ,  Defense  Acquisition  Management  Information 
Retrieval  (DAMIR)  System,  December  2003,  7. 


10 


increase  was  not  driven  by  increased  capability,  however,  but  by  the  unrealistic  cost 
estimate  in  the  MS  B  SBIRS-High  baseline.20 

In  the  GBS  example,  it  seems  clear  that  capabilities  beyond  those  in  the  MS  B 
baseline  were  added  to  the  program.  While  unit  cost  did  increase,  that  was  a  matter  of 
paying  more  for  more.  For  SBIRS-High,  in  contrast,  it  appears  that  the  advent  of  a  boom 
funding  climate  provided  a  program  experiencing  severe  problems  an  opportunity  to  “get 
well.”  In  effect,  in  such  cases,  what  otherwise  would  have  been  capability  shortfalls  were 
converted  into  cost  growth  and,  relative  to  MS  B,  DoD  eventually  paid  more  for  the  MS 
B  capability  than  had  been  anticipated.  The  boom  effect  includes  both  of  these  cases.  So 
does  accretion  of  PAUC  growth  during  bust  years. 

The  average  PAUC  growth  of  all  DS ARC/DAB  bust  programs  is  38  percent. 
Without  making  a  specific  estimate,  P-5126  suggested  that  most  of  this  PAUC  growth 
stemmed  from  flawed  MS  B  baselines.  In  the  language  of  the  present  paper,  if  all  of  the 
unit  cost  growth  actually  is  a  matter  of  “getting  well,”  the  PAUC  growth  due  to  flawed 
MS  B  baselines  problems  remains  38  percent.  It  is  less  than  38  percent  to  the  extent  that 
PAUC  growth  of  MDAPs  in  Bustl,  in  both  the  years  they  spent  in  bust  climates  and  they 
spent  in  boom  climates,  is  due  to  decisions  to  acquire  capabilities  beyond  those  of  the  MS 
B  baselines.  Parts  of  PAUC  growth  in  years  spent  in  both  boom  and  bust  climates  post 
MS  B  very  probably  do  reflect  acquisition  of  capabilities  beyond  that  of  the  MS  B 
baseline.  Unfortunately,  we  do  not  have  a  way  to  differentiate  between  PAUC  growth 
due  to  acquisition  of  additional  capability  and  that  due  to  an  increase  in  the  actual  costs  of 
the  MS  B  capability.  Further  statistical  analysis  along  the  lines  of  that  presented  here 
seems  unlikely  to  be  useful  in  untangling  these  two  elements.  Instead,  progress  on  the 
question  of  why  some  programs  but  not  others  in  Bustl  experienced  a  boom  effect 
probably  will  require  detailed  examination  of  changes  in  the  relevant  programs  post 
MSB. 


20 

“  See  Gene  Porter  et  at.,  “The  Major  Causes  of  Cost  Growth  in  Defense  Acquisition:  Volume  I-Executive 
Summary,”  IDA  Paper  P-4531  (Alexandria,  VA:  Institute  for  Defense  Analyses,  December  2009), 
especially  ES-29.  See  also  Obaid  Younossi  et  al.,  “Improving  the  Cost  Estimation  of  Space  Systems,” 
MG-690-AF  (Santa  Monica,  CA:  The  RAND  Corporation,  2008),  26-27;  and  Yool  Kim  et  ah, 
“Acquisition  of  Space  Systems,  Volume  7:  Past  Problems  and  Future  Challenges,”  RAND  MG-1171/7- 
OSD  (Santa  Monica,  CA:  The  RAND  Corporation,  2015),  7. 
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Appendix  A. 
Background 


This  appendix  provides  brief  descriptions  of  (1)  the  approach  taken  in  this  paper  to 
the  examination  of  unit  cost  growth;  (2)  each  of  the  acquisition  policy  and  process 
periods  used,  and  (3)  the  funding  climates.  The  material  is  drawn  mainly  from  Institute 
for  Defense  Analyses  (IDA)  papers  P-5 126,  P-52 18,  and  P-5330  (Revised). 1 

A.  Approach 

Binning  cost  growth  in  terms  of  budget  climate  and  acquisition  policy  and  process  is 
a  departure  from  other  statistical  studies  of  cost  growth  and,  for  that  reason,  warrants 
brief  discussion.  Statistical  studies  on  cost  growth  in  individual  Major  Defense 
Acquisition  Programs  (MDAPs)  typically  assume  that  most  cost  growth  can  be  explained 
by  characteristics  of  the  program.  The  objective  of  these  studies  is  to  link  program 
characteristics  to  cost  growth  and  thereby  provide  a  basis  for  identifying  program 
characteristics — program  “do’s  and  do  not’s” — that  will  promote  program  success  in 
terms  of  cost,  schedule,  and  performance. 

This  paper,  in  contrast,  does  not  attempt  to  explain  why  cost  growth  occurs  but 
instead  when  it  occurs,  in  chronological  time  and  in  the  acquisition  cycle,  and  where  it 
occurs  in  terms  of  Department  of  Defense  (DoD)  processes.  The  analysis  takes  it  as  a 
point  of  departure  that  the  proximate  causes  of  Program  Acquisition  Unit  Cost  (PAUC) 
growth  are  decisions  embedded  in  programs  approved  at  Milestone  (MS)  B  (unrealistic 
cost  estimates  or  programmatic  assumptions,  for  example)  and  decisions  made  during 
program  execution  (such  as  failing  to  act  promptly  enough  on  test  results)  that  eventually 
caused  the  PAUC  growth.  With  this  assumption  in  the  background,  this  paper  and  the 
others  in  the  series  in  effect  ask  whether  factors  that  cause  (for  example)  cost  growth  tend 
to  cluster  in  certain  circumstances. 


1  David  L.  McNicol  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy  and  Process  on 
Cost  Growth  of  Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5126  (Alexandria,  VA:  Institute 
for  Defense  Analyses,  September  2014);  David  L.  McNicol,  Sarah  K.  Burns,  and  Linda  Wu,  “Evidence 
on  the  Effect  of  DoD  Acquisition  Policy  and  Process  and  Funding  Climate  on  Cancellation  of  Major 
Defense  Acquisition  Programs,”  IDA  Paper  P-5218  (Alexandria,  VA:  Institute  for  Defense  Analyses, 
May  2015);  and  David  L.  McNicol  et  at,  “Further  Evidence  on  the  Effect  of  Acquisition  Policy  on  Cost 
Growth  of  Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5330  (Revised)  (Alexandria,  VA: 
Institute  for  Defense  Analyses,  August  2016). 
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The  list  of  factors  that  can  be  expected  to  cause  such  clustering  is  short.  The  first  of 
these  is  acquisition  policy  and  process.  For  example,  acquisition  reforms  adopted  in  mid- 
1969  strongly  discouraged  the  use  of  fixed  price  contracts  for  development  of  MDAPs, 
and  put  in  place  a  process  for  oversight  of  compliance  with  this  and  other  acquisition 
policies.  The  second  is  funding  climate,  on  the  expectation  that  unrealistic  or  unduly 
optimistic  assumptions  are  more  likely  to  be  embodied  in  the  baselines  of  MDAPs  that 
pass  MS  B  in  bust  funding  climates. 

B.  Acquisition  Policy  and  Process  Periods 

The  President  of  the  United  States  or  the  Secretary  of  Defense,  from  DoD’s  early 
days,  occasionally  cancelled  or  initiated  major  acquisitions.  MDAPs  were  also  subject  to 
review  during  the  budget  cycle  by  the  Office  of  the  Assistant  Secretary  of  Defense 
(Comptroller)  and  the  Office  of  Management  and  Budget.  From  the  creation  of  the 
National  Security  Establishment  in  1947  through  1960,  however,  the  Office  of  the 
Secretary  of  Defense  (OSD)  had  no  institutionalized  process  for  the  oversight  of  major 
weapon  system  acquisitions. 

The  first  steps  towards  installing  an  OSD-level  acquisition  oversight  process  were 
taken  by  then  Secretary  of  Defense  Robert  McNamara  in  1964.  There  have  since  been 
many  dozens  of  changes  in  acquisition  policy  or  process  made  by  senior  OSD  officials 
and  on  the  order  of  a  dozen  major  changes  directed  by  the  Congress. 

We  organize  the  large  number  of  policy  and  process  changes  by  identifying  time 
periods  during  which  the  main  features  of  acquisition  policy  and  process  remained 
approximately  the  same.  These  are  as  follows:  2 

1.  The  McNamara-Clifford  years,  FY  1964-FY  1969 

2.  The  Defense  Systems  Acquisition  Review  Council  (DSARC),  FY  1970- 
FY  1982 

3.  The  Post-Carlucci  Initiatives  DSARC,  FY  1983-FY  1989 

4.  The  Defense  Acquisition  Board  (DAB),  FY  1990-FY  1993 

5.  Acquisition  Reform  (AR),  FY  1994-FY  2000 

6.  The  DAB  Post-AR,  FY  2001  to  date 


2 

"  These  categories  are  abstracted  from  J.  Ronald  Fox,  Defense  Acquisition  Reform,  1969  to  2009 :  An 
Elusive  Goal  (Washington,  DC:  U.S.  Army  Center  of  Military  History,  2011). 
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1.  McNamara-Clifford,  1964-1969 

The  best  known  of  McNamara’s  changes  in  OSD  decision-making  processes  was 
the  creation  of  the  Office  of  Systems  Analysis  (OSA)  and  the  Planning,  Programming, 
and  Budgeting  System  (PPBS),  which  moved  decisions  on  what  major  systems  to  procure 
to  the  Secretary  of  Defense  level.  OSA  and  PPBS  so  overshadowed  development  of  an 
OSD-level  acquisition  oversight  process  that  the  latter  is  no  longer  part  of  the  collective 
memory  of  the  DoD  acquisition  community.  Nonetheless,  the  evolution  of  the  OSD-level 
acquisition  process  began  with  initiatives  taken  by  McNamara  in  the  mid-1960s. 

Provisions  for  milestone  reviews  appeared  in  1964  with  the  issuance  of  DoD 
Directive  (DoDD)  3200.9,  Initiation  of  Engineering  and  Operational  Systems 
Development ,3  This  original  version  of  the  directive  set  one  point  at  which  OSD — in 
principle,  the  Secretary  of  Defense — approval  was  required  for  an  acquisition  program  to 
proceed.  In  1965,  a  second  decision  point  was  added. 

The  two  milestones  of  the  revised  DoDD  3200.9  defined  an  acquisition  cycle  with 
three  phases.  The  first  of  these  “was  called  concept  formulation.  During  concept 
formulation  OSD  and  the  Service(s)  involved  assured  themselves  that  they  were  buying 
the  right  system  to  meet  real  needs  and  that  the  technology  was  fully  ready.”4  Concept 
formulation  typically  was  initiated  by  a  Service,  but  involved  the  Director,  Defense 
Research  and  Engineering  (DDR&E)  and  the  Office  of  the  Assistant  Secretary  of  Defense 
for  Systems  Analysis  (OASD(SA)),  and  included  what  would  now  be  called  an  Analysis 
of  Alternatives,  led  by  OASD(SA).  It  also  apparently  included  what  would  later  be  called 
a  Mission  Element  Need  Statement  and  also  the  main  parts  of  an  Acquisition  Strategy 
and  plans  for  oversight  of  the  program  as  it  proceeded. 

Approval  to  proceed  from  the  Concept  Fonnulation  phase  authorized  the  Service 
sponsoring  the  program  to  fund  at  least  one  company  to  prepare  a  definitized  contract 
proposal.  Preparation  of  a  definitized  contract  was  the  second  phase  of  the  acquisition 
process.  The  third  phase  was  the  award  of  a  contract  for  development  and  procurement  of 
the  system.  That  is  to  say,  the  second  of  DoDD  3200. 9’s  milestones  combined  what  now 
would  be  called  MS  B  and  MS  C  authority. 

The  1965  revision  of  DoDD  3200.9  stated  that  one  of  the  purposes  of  the  Concept 
Formulation  phase  was  to  “[pjrovide  a  basis  for  a  firm  fixed  price  or  fully  structured 


3 

The  first  version  of  DoDD  3200.9  was  issued  in  1964.  A  revision  that  made  provision  for  the  Contract 
Definition  Phase  was  issued  July  1,  1965.  See  Thomas  K.  Glennan,  Jr.,  “Policies  for  Military  Research 
and  Development,"  RAND  Paper  P-3253  (Santa  Monica  CA:  The  RAND  Corporation,  1966),  12. 

4  William  D.  O’Neil  and  Gene  H.  Porter,  “What  to  Buy?  The  Role  of  Director  of  Defense  Research  and 
Engineering  (DDR&E) — Lessons  from  the  1970s,”  IDA  Paper  P-4675  (Alexandria,  VA:  Institute  for 
Defense  Analyses,  January  2011),  30. 
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incentive  contract  for  Engineering  Development.”5  In  addition,  McNamara  directed  the 
use  of  Total  Package  Procurement  (TPP)  when  it  was  judged  to  be  practicable  and,  when 
not,  a  Fixed  Price  Incentive  Fee  (FPIF)  or  Cost  Plus  Incentive  Fee  (CPIF)  contract.  By 
1966,  McNamara  had  concluded  that  TPP  contracts  were  in  fact  not  a  practicable  way  to 
acquire  most  major  weapon  systems,6  although  acquisition  policy  apparently  still  had  a 
tilt  towards  fixed  price  contracts,  even  for  development. 

The  OSD-level  milestone  reviews  were  structured  around  what  was  then  called  the 
Development  Concept  Paper  (DCP).  The  requirement  for  a  DCP  was  instituted  in  1965 
by  the  DDR&E,7  and  in  1968,  it  was  required  in  order  to  initiate  any  major  development 
project.  DDR&E  coordinated  initial  DCPs  with  concerned  OSD  offices  (and  probably  the 
Joint  Staff  and  other  Services)  and  with  what  now  would  be  called  the  Milestone 
Decision  Authority  (MDA)  for  the  initial  DCP.8  Once  approved  by  DDR&E,  the 
proposed  new  start  went  to  the  Secretary  of  Defense,  although  the  sources  consulted  do 
not  indicate  whether  it  went  as  a  separate  action  or  as  part  of  the  Service’s  budget 
submission.  It  is  also  not  clear  which  OSD  official  was  the  MDA  for  the  second 
milestone. 

2,  The  1969  Packard  Reforms 

The  start  of  the  second  acquisition  period  is  marked  by  reforms  initiated  by  then 
Deputy  Secretary  of  Defense  David  Packard  in  mid-1969.  The  elements  of  the  Packard 
reforms  fall  into  three  main  categories.  First,  Packard  clarified  and  fonnalized  the  OSD- 
level  acquisition  oversight  process  that  had  emerged  under  McNamara  and  Clifford.  This 
was  achieved  through  a  combination  of  steps: 

•  The  Defense  Systems  Acquisition  Review  Council  (DSARC)  was  established  to 
advise  the  MDA  at  each  milestone. 

•  DDR&E  was  designated  as  the  MDA  at  MS  I  and  MS  II;  the  Assistant  Secretary 
of  Defense  for  Installations  and  Logistics  was  the  MDA  for  MS  III. 


5  DoDD  3200.9,  Initiation  of  Engineering  and  Operational  Systems  Development,  July  1,  1965,  V.B(l). 

6  Gordon  Adams,  Paul  Murphy,  and  William  Grey  Rosenau,  Controlling  Weapons  Costs:  Can  Pentagon 
Reforms  Work?  (New  York:  Council  on  Economic  Priorities,  1983),  19-20.  A  TPP  contract  is  one  that 
covers  EMD,  at  least  a  significant  portion  of  procurement,  and  at  least  part  of  the  support  of  the  system 
(for  example,  depot  maintenance).  Fox  also  notes  that  McNamara  moved  to  consolidate  acquisition 
functions  in  defense  agencies — e.g.,  the  agency  that  became  the  Defense  Logistics  Agency — and 
promoted  the  use  by  program  managers  of  particular  management  tools  such  as  PERT  and  earned  value. 

7  Fox,  Defense  Acquisition  Reform,  1969  to  2009,  24-47,  provides  a  sketch  of  how  the  process  evolved 
and  worked  during  the  1960s. 

8 

See  C.  W.  Borklund,  The  Department  of  Defense  (New  York:  Frederick  A.  Praeger,  1969),  83. 
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•  The  Development  Concept  Paper  was  renamed  the  Decision  Coordinating  Paper 
(retaining  the  acronym). 

•  Acquisition  policy  and  the  process  of  OSD-level  milestone  and  program  reviews 
were  laid  out  in  a  new  directive  (DoDD  5000.1)  and  a  new  top-level  process 
instruction  (DoDI  5000.2). 9 

As  each  of  these  steps  had  antecedents  in  the  McNamara-Clifford  process,  it  seems 
reasonable  to  say  that  they  were  evolutionary  rather  than  revolutionary. 10 

Second,  Packard  made  changes  to  policy  on  contract  types.  Picking  up  on  this  topic 
where  McNamara  had  left  off,  he  ruled  out  the  use  of  TPP  and  discouraged  the  use  of 
FPIF  for  development  contracts  in  favor  of  CPIF.  (Cost  Plus  Award  Fee  may  not  have 
been  included  in  the  contracting  play  book  yet).  As  a  general  matter,  Packard’s  policy 
was  to  match  contract  terms  to  the  riskiness  of  the  acquisition. 

Third,  and  finally,  Packard  redefined  the  milestones  so  as  to  separate  the  decision  to 
allow  the  program  to  enter  Engineering  and  Manufacturing  Development  (EMD)  (now 
MS  B)  from  the  decision  to  enter  the  Production  phase  (now  MS  C)  and  required 
OSD-level  approval  of  each  decision.  He  also  collapsed  DoDD  3200. 9’s  Contract 
Definition  phase  into  the  new  and  broader  Validation  phase,  which  has  at  various  times 
since  been  called  Demonstration  and  Validation,  Program  Development  and  Risk 
Reduction  and,  currently,  Technology  Maturation  and  Risk  Reduction.  MS  I  (now  MS  A) 
authorized  entry  into  this  phase.  These  changes  were  more  revolutionary  than 
evolutionary. 1 1 

Packard’s  reforms  were  at  the  time  explained  as  a  return  to  the  Components  of  the 
authority  to  manage  major  acquisition  programs.  Compared  to  cases  such  as  the  F-l  1 1,  in 
which  McNamara  played  a  very  active  role  making  decisions  about  the  program,  it  was. 
Under  the  new  acquisition  directives,  the  Secretary  of  Defense,  while  retaining  full  legal 
authority  over  acquisition  programs,  would  act  through  the  established  acquisition 
process  except  in  extraordinary  circumstances.  Decisions  at  the  DSARC  level  were 
advisory  to  the  Secretary  and  Deputy  Secretary  of  Defense  but,  apart  from  exceptional 
cases,  they  probably  reached  that  level  by  way  of  the  Service’s  proposed  budgets  (and  the 
Comptroller  was  the  backstop  enforcer  of  the  requirement  for  milestone  approval  before  a 
program  could  advance  to  the  next  stage).  In  comparison  to  cases  such  as  the  F-l  1 1,  this 


9  DoDD  5000.1,  The  Defense  Acquisition  System,  May  12,  2003;  and  DoDI  5000.2,  Operation  of  the 
Defense  Acquisition  System,  May  12,  2003. 

10  Fox,  Defense  Acquisition  Reform,  1969  to  2009,  57,  provides  a  useful  schematic  comparison  of  the 
DoDD  3200.9  milestones  and  those  of  Packard’s  DoDD  5000.1/DoDI  5000.2. 

1 1  DoDI  5000.2,  issued  October  23,  2000,  formally  established  MSs  A,  B,  and  C  (in  place  of  MSs  I,  II, 
and  III)  as  the  main  decision  points  for  an  MDAP.  The  definitions  are  such  that  MS  B  is  placed  several 
months  earlier  in  the  process  than  MS  II. 
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implied  less  OSD-level  control  over  major  acquisitions.  The  DSARC,  however,  had  a 
greater  substantive  scope  than  the  1960s  process  for  the  more  typical  program  and  was 
more  tightly  organized.  For  the  large  majority  of  major  acquisition  programs,  then,  the 
new  DSARC  process  probably  was  more  effective. 12 

The  process  instituted  by  Packard  in  mid- 1969  had  all  of  the  basic  features  that  are 
still  central  to  OSD-level  oversight  of  MDAPs — three  milestones,  with  definitions  similar 
to  those  used  now;  formal  milestone  reviews;  a  body  to  advise  the  MDA;  and  a 
document(s)  that  described  the  basic  features  of  the  program,  provided  a  vehicle  for  staff 
inputs,  and  set  down  the  cost,  schedule,  and  perfonnance  goals  that  the  program  was  to 
meet. 13  This  is  an  important  point  because  it  implies  that  the  effects  of  Packard’s  reforms 
would  endure,  so  long  as  the  process  was  not  altered  in  a  fundamental  way. 

3.  The  More  Recent  Periods 

The  vehicle  for  the  transition  from  the  first  phase  of  the  DSARC  (FY  1970- 
FY  1982)  to  the  second  (FY  1983-FY  1989)  was  the  Carlucci  Initiatives,  named  after 
then  Deputy  Secretary  of  Defense  Frank  Carlucci.  These  were  developed  during  calendar 
year  (CY)  1981  and  implemented  during  FY  1982-FY  1983.  The  Carlucci  Initiatives  did 
not  involve  any  major  changes  in  the  DSARC  process  or  in  the  policies  Packard  had 
established;  in  these  terms,  the  Carlucci  Initiatives  were  more  reaffirmation  than  change. 
The  Initiatives,  however,  included  measures  designed  to  coordinate  decisions  on  MDAPs 
made  in  the  PPBS  and  those  made  in  the  DSARC  process.  Important  statutory  changes 
were  also  made  during  1982  and  1983. 


12 

~  Clark  A.  Murdock,  Defense  Policy  Formation  (Albany,  NY :  State  University  of  New  York  Press, 

1974),  155-179,  disagrees  with  this  judgment.  Murdock  is  primarily  concerned  with  Systems  Analysis 
and  resource  allocation,  but  also  comments  specifically  on  the  acquisition  process.  In  particular,  he 
notes  that  the  new  Decision  Coordinating  Paper  did  not  provide  “any  mechanism  for  ongoing 
managerial  control.”  This  is  accurate  in  that  the  Packard  reforms  placed  management  of  the  programs  in 
the  hands  of  the  Services.  It  is  incomplete  in  that  the  Services  were  responsible  for  staying  within  what 
would  later  be  called  the  Acquisition  Program  Baseline,  and  the  MDA  was  enjoined  to  act  in  cases  in 
which  they  did  not. 

13 

In  January  1972,  the  DSARC  process  was  expanded  to  include  an  independent  cost  estimate  at  MS  II 
and  MS  III  provided  by  the  Cost  Analysis  Improvement  Group  (CAIG),  newly  established  by  a 
memorandum  signed  by  Melvin  Laird  on  January  25,  1972.  A  December  7,  1971  memorandum  signed 
by  David  Packard  directed  the  Military  Departments  to  begin  using  “independent  parametric  cost 
analysis.”  See  Donald  Srull,  ed..  The  Cost  Analysis  Improvement  Group:  A  History  (McLean,  VA: 
Logistics  Management  Institute  (now  LMI),  1998),  47-48.  Since  the  implementation  of  the  Weapon 
Systems  Acquisition  Reform  Act  of  2009  (WSARA),  the  independent  cost  estimates  have  been 
provided  by  the  Cost  Assessment  Deputate  of  the  Office  of  Cost  Assessment  and  Program  Evaluation 
(CAPE). 
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The  DSARC  was  followed  by  the  DAB.14  While  the  DAB  itself  bears  a  strong 
family  resemblance  to  the  DSARC,  the  statute  that  created  it  also  created  the  position  that 
is  now  called  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
(USD(AT&L))  and  Service  Acquisition  Executives  who  reported  to  the  USD(AT&L); 
designated  USD(AT&L)  as  the  MDA  for  most  MDAPs;  and  removed  the  Service  chiefs 
from  the  acquisition  chain  of  command.  In  addition,  the  statute  created  the  position  of 
Vice  Chairman  of  the  Joint  Chiefs  of  Staff  (VCJCS)  and  prompted  a  new  requirements 
process  centered  on  the  VCJCS. 

AR  (FY  1994-FY  2000)  was  to  a  large  extent  intended  to  put  acquisition  of  major 
weapon  systems  on  a  more  commercial  basis  and  make  it  easier  for  firms  outside  the 
defense  sector  to  sell  to  DoD.  Contracts  were  structured  so  that  defense  contractors 
assumed  more  responsibility  for  system  performance;  correspondingly,  DoD’s  role  in 
contract  management  decreased.  OSD  oversight  of  MDAPs  also  was  relaxed  somewhat. 
Substantial  cuts  in  acquisition  staffs  at  both  the  OSD  level  and  Service  Headquarters 
level  were  made,  and  senior  decision  makers  took  a  more  pennissive  attitude  towards 
cost  growth. 

The  post-AR  DAB  period  was  marked  by  the  arrival  of  a  new  administration  in 
January  2001.  This  period  saw  no  overt  rejection  of  AR,  but  most  of  its  initiatives  were 
no  longer  pursued.  The  new  administration  seemed  implicitly  to  favor  a  return  to  the 
status  quo  ante  in  OSD-level  oversight  of  acquisition  programs. 

In  2006,  the  Congress  strengthened  the  Nunn-McCurdy  Act,  which  had  been  passed 
in  1982. 15  In  2009,  the  Congress  passed  the  Weapon  Systems  Acquisition  Reform  Act 
(WSARA),  which  made  several  changes  that  may  prove  to  be  consequential. 

C.  Funding  Climates 

It  is  important  to  realize  that  funds  appropriated  for  procurement  are  a  lagging 
indicator  of  a  change  in  budget  climate.  Funding  and  programmatic  decisions  embedded 
in  the  MS  B  baselines  of  MDAPs  are  made  at  least  one  or  two  years  before  the 
President’s  Budgets  in  question  are  submitted  to  the  Congress.  Consequently,  those 
decisions  necessarily  reflect  expectations  held  by  decision  makers  about  the  future  DoD 
budget  climate,  and  the  breakpoints  between  different  budget  climates  should  mark  the 
points  at  which  there  were  major  shifts  in  expectations. 


14  There  is  some  uncertainty  about  when  the  Post-Carlucci  Reforms  DSARC  should  end  and  the  DAB 
period  should  begin.  The  relevant  statutes  were  passed  in  December  1985  (first  quarter  FY  1986),  and 
the  DAB  began  functioning  under  that  name  in  late  FY  1987  or  early  FY  1988;  however,  DoD  did  not 
implement  the  full  set  of  reforms  required  by  statute  until  1990.  We  have  for  that  reason  set  the  line  at 
1990. 

15  The  Nunn-McCurdy  Act  imposes  reporting  and  other  requirements  that  senior  DoD  officials  must  meet 
if  an  MDAP  breaches  certain  unit  cost  thresholds. 
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It  is  also  important  to  recognize  the  distinction  between  the  development  of  the  DoD 
budget  request  for  acquisition  and  the  amount  the  Congress  eventually  appropriates.  The 
Components’  acquisition  programs  are  built  subject  to  funding  constraints  specified  by 
senior  Component  officials  (in  particular,  the  Service  Secretaries)  based  on  fiscal 
guidance  they  are  given  by  the  Secretary  of  Defense.  It  is  reasonable  to  assume  that 
expectations  about  future  funding  levels  are  formed  as  much  by  “this  year’s”  fiscal 
guidance  as  by  “last  year’s”  congressional  appropriation. 

We  used  four  events  to  identify  the  breakpoints  between  funding  climates:  (1)  the 
invasion  of  Afghanistan  by  the  USSR  in  late  December  1979;  (2)  passage  of  the  Gramm- 
Rudman-Hollings  Act  in  December  1985;  (3)  the  terrorist  attack  on  the  United  States  on 
September  11,  2001;  and  (4)  the  start  of  the  withdrawal  of  US  forces  from  Iraq  in  late 
FY  2009.  Senior  decision  makers  could  reasonably  expect  each  of  the  events  identified  to 
signal  major  and  sustained  changes  in  the  defense  funding  climate,  which  in  fact  they  did. 

1.  The  First  Bust  Climate 

DoD  procurement  funding  increased  markedly  during  the  first  three  years  of  the 
Kennedy  administration  (CY  1961-CY  1963)  and  remained  relatively  high  through  the 
1960s.  A  large  part  of  the  increase  in  procurement  funding  during  FY  1964-FY  1969, 
however,  was  for  munitions  and  procurement  to  replace  systems  lost  in  combat, 
particularly  aircraft.  Insofar  as  modernization  of  weapon  systems  was  concerned,  there 
was  little  or  no  boom  associated  with  the  Vietnam  War. 16 

Gerald  Ford  was  sworn  in  as  president  on  August  9,  1974,  about  six  weeks  into 
FY  1975.  The  FY  1976  budget,  preparation  of  which  was  far  advanced  when  Ford 
became  president,  increased  procurement  by  13.5  percent.  The  amount  appropriated  for 
procurement  in  FY  1977  increased  by  an  additional  30  percent,  and  the  amounts 
appropriated  for  procurement  during  the  next  two  years  were  only  modestly  lower. 

There  are  three  main  reasons  for  concluding  that  the  FY  1976  and  FY  1977 
increases  did  not  cause  a  major  shift  in  expectations  about  defense  spending.  First,  they 
basically  reversed  sharp  declines  in  procurement  funding  during  the  years  1970-1975, 
roughly  the  period  of  the  “Peace  Dividend”  that  had  accompanied  the  end  of  the  War  in 
Vietnam. 

Second,  there  was  no  political  consensus  in  favor  of  a  sustained  substantial  increase 
in  funding  for  DoD  procurement.  As  evidence  of  this  point,  during  the  1976  campaign, 


16  These  comments  are  based  on  an  unpublished  IDA  working  database  drawn  from  various  US 
government  sources.  We  are  indebted  to  Dr.  Daniel  Cuda  for  providing  these  data. 
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President  Carter  “pledge[d]  to  reduce  the  defense  budget  by  $7  billion  and  submit  a 
balanced  budget  in  fiscal  year  1981.” 17 

Third,  while  Ford  opposed  cuts  to  defense  spending,  the  material  consulted  for  this 
paper  (Ford’s  three  State  of  the  Union  addresses  and  campaign  material)  did  not  indicate 
that  he  attempted  to  establish  a  consensus  favoring  large  increases  in  DoD  spending.  It 
also  is  far  from  clear  that  he  was  in  a  position  to  do  so  during  most  of  his  presidency.  The 
Democrats  controlled  both  houses  of  the  Congress.  In  his  inaugural  address  to  the 
Congress,  Ford  said:  “I  am  your  man,  for  it  was  your  carefully  weighted  confirmation 
that  changed  my  occupation.”  That  is,  his  legitimacy  as  president  was  derived  from  the 
Congress  rather  than  from  election  by  the  people. 

The  Carter  administration  followed  through  on  this  campaign  commitment  by 
reducing  the  FY  1978  DoD  budget  request  left  by  the  Ford  administration  by  about  $3 
billion.  Moreover,  on  August  17,  1978,  President  Carter  vetoed  the  1979  Defense 
Authorization  Act.  The  veto  message  implies  that  the  Congress  had  put  a  $2  billion 
nuclear  powered  carrier  into  the  budget  Carter  had  submitted  from  the  Ford 
administration  and  cut  some  other  procurement  programs,  some  readiness  funding,  and 
some  RDT&E  funding.  Carter  wanted  the  carrier  removed  from  the  budget  and  the  other 
funding  he  had  requested  restored.  The  veto  was  sustained.  As  these  facts  suggest  would 
be  the  case,  DoD  Budget  Authority  (BA)  was  lower  for  FY  1978  and  FY  1979  than  it  was 
for  FY  1977,  and  FY  1980  was  only  slightly  larger. 

2.  Start  of  the  First  Boom  Climate 

The  Soviet  invasion  of  Afghanistan  in  late  December  1979  (three  months  into 
FY  1980)  prompted  a  change  in  the  Carter  administration’s  policy  on  defense  spending. 
One  source  states  that  “only  a  month  after  the  Soviet  invasion  of  Afghanistan,  Carter 
called  for  increases  in  defense  spending  of  4.6  percent  per  year,  every  year  over  five 
years.”18  Carter’s  State  of  the  Union  address  delivered  on  January  21,  1980,  listed  as  one 
of  his  few  new  proposals  “Initiatives  implementing  my  response  to  the  Soviet  invasion  of 
Afghanistan.”  The  “Defense  Spending”  portion  of  the  State  of  the  Union  address  stated 
that  requested  FY  1981  funding  authority  was  a  more  than  5  percent  real  increase  over 
the  funding  requested  for  FY  1980.  It  goes  on  to  characterize  this  as  “a  growth  rate  for 
defense  that  we  can  sustain  over  the  long  haul.”  In  fact,  the  real  increase  in  enacted  DoD 


17  «  >> 

Frank  L.  Jones,  A  "Hollow  Army  ”  Reappraised:  President  Carter,  Defense  Budgets,  and  the  Politics  of 
Military’  Readiness  (Carlisle,  PA:  Strategic  Studies  Institute,  U.S.  Army  War  College,  2012),  16. 

18 

Andrew  Krepinevich,  “What  Would  Jimmy  Carter  Do?,”  Defense  One,  March  1 1,  2014, 
http://www.defenseone.com/ideas/2014/03/what-would-jimmy-carter-do/80335/7oreUsearch_ 
what%2  0  would%2  Oj  immy  %2  0carter%20do . 
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BA  for  FY  1981  was  12  percent  for  DoD  as  a  whole  and  about  25  percent  for 
procurement. 

President  Reagan,  who  took  office  in  January  1981,  requested  increases  in  DoD 
funding  that  went  well  beyond  what  the  Carter  administration  had  planned.  By  FY  1985, 
DoD  funding  for  procurement  was  more  than  70  percent  above  what  it  had  been  in 
FY  1981. 

3.  The  Second  Bust  Period 

Congressional  appropriations  for  defense  procurement  peaked  in  FY  1985.  The 
Congress  did  not  enact  the  increase  in  DoD  funding  for  FY  1986  requested  by  the 
administration,  and  in  December  1985  passed  the  Gramm-Rudman-Hollings  (GRH)  Act, 
which  provided  for  sequestration  to  satisfy  budget  targets. 

None  of  this  was  known,  however,  when  the  FY  1986  DoD  budget  request  was  sent 
to  the  Congress  in  late  January  or  early  February  1985,  and  the  defense  buildup  was  in 
full  swing  during  FY  1983  through  FY  1985  when  the  FY  1986  request  was  being 
developed.  For  that  reason,  this  paper  takes  FY  1986  as  the  last  year  of  the  Carter-Reagan 
boom  climate. 

The  Congress  cut  DoD  funding  for  FY  1987  about  3  percent  below  the  FY  1986 
level.  Cuts  in  defense  spending  continued  through  the  end  of  the  Reagan  administration. 
Procurement  funding  for  FY  1987  was  down  about  16  percent  from  its  FY  1986  level  and 
dropped  modestly  further  over  the  next  three  fiscal  years.  Additional  large  drops  came 
after  the  fall  of  the  Berlin  Wall  on  November  9,  1989  (first  quarter  FY  1990)  and  the 
collapse  of  the  Soviet  Union  on  December  25,  1991  (first  quarter  FY  1992).  By  the  mid- 
1990s,  DoD  procurement  funding  had  returned  to  about  the  post-Vietnam  War  levels  of 
the  mid-1970s  and  was  little  more  than  one-third  of  its  peak  (FY  1985)  level.  Altogether, 
the  second  bust  phase  continued — for  a  total  of  16  years — through  FY  2002. 

4.  The  Second  Boom  Period 

The  defense  spending  boom  that  followed  was  prompted  by  the  A1  Qaeda  attacks  of 
September  11,  2001  (late  FY  2001)  and  the  subsequent  wars  in  Afghanistan  and  Iraq.  The 
question  is  where  the  start  of  the  boom  climate  should  be  placed — at  FY  2001,  FY  2002, 
or  FY  2003? 

George  W.  Bush  was  inaugurated  on  January  20,  2001.  It  is  not  clear  that  the 
outgoing  Clinton  administration  submitted  its  final  (FY  2002)  budget  to  the  Congress. 
The  Bush  administration  apparently  submitted  a  pro  fonna  budget  (A  Blueprint  for  New 
Beginnings)  on  February  29,  2001,  and  a  complete  FY  2002  request  proposal  on  April  9. 
The  Bush  administration  also  requested  a  $5.2  billion  increase  in  the  FY  2001 
appropriation.  According  to  the  budget  news  release,  this  was  primarily  for  Military 
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Personnel  (MILPERS)  and  Operations  and  Maintenance  (O&M).  As  a  candidate,  Bush 
had  not  committed  to  increasing  procurement,  and  his  proposed  budget  for  FY  2002  did 
not  do  so.  In  fact,  procurement  was  actually  $1.4  billion  lower  than  the  FY  2002  column 
of  the  FY  2001  Clinton  Future  Years  Defense  Plan. 

Post  9/11,  the  budget  climate  was  radically  different.  On  December  28,  2001  (end  of 
the  first  quarter  of  FY  2002),  the  Congress  adopted  an  emergency  supplemental  for 
FY  2001,  even  though  it  had  ended  three  months  earlier,  and  on  January  10,  2002 
adopted  an  emergency  supplemental  for  FY  2002.  The  spending  plans  that  went  into  the 
proposed  FY  2002  budget  could  not  have  been  influenced  by  these  events. 

Planning  for  the  FY  2004  DoD  budget  certainly  took  place  in  a  boom  climate.  That 
is  less  clearly  true  of  the  FY  2003  budget.  The  FY  2003  DoD  budget  submitted  February 
4,  2002  was  about  10  percent  above  the  FY  2002  budget  request  and,  more  important  for 
present  purposes,  provided  for  about  a  10  percent  increase  in  procurement  funding. 
Barely  four  months  elapsed  between  9/11  and  the  submission  of  the  FY  2003  DoD 
budget.  On  the  supposition  that  major  changes  in  acquisition  programs  could  have  been 
made  within  that  amount  of  time,  this  paper  places  the  start  of  the  post-9/ 1 1  boom  climate 
at  FY  2003. 

DoD  funding  for  procurement  increased  in  FY  2003  by  about  20  percent  over  the 
preceding  year  and  continued  to  increase  through  FY  2008.  FY  2009  probably  should  be 
counted  as  the  last  year  of  this  second  boom  period  because  the  Great  Recession  began  in 
its  first  quarter  and  the  withdrawal  of  US  troops  from  Iraq  began  in  July  of  2009. 
Expectations  of  a  decline  in  defense  spending  developed  at  that  time,  and  were  solidified 
by  passage  of  the  Budget  Control  Act  of  2011,  which  was  signed  into  law  in  August 
2011. 
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Appendix  B 
The  Data 


A  compact  disc  (CD)  in  a  pocket  on  the  inside  back  cover  of  this  paper  contains 
the  database  used  in  this  research.  The  CD  also  contains  several  other  files  that  help  to 
document  or  explain  the  data.  This  appendix  provides  background  infonnation  that 
facilitates  access  to  and  use  of  the  files  on  the  CD. 

A.  Cost  Growth  Metric  and  Ground  Rules 

The  cost  growth  metric  used  in  this  paper  is  quantity  normalized  growth  in  Program 
Acquisition  Unit  Cost  (PAUC)  in  program  base  year  dollars.  In  most  instances  the  PAUC 
growth  figure  used  is  measured  from  the  MS  B  baseline.  PAUC  includes  Research, 
Development,  Test,  and  Evaluation  (RDT&E)  funding  as  well  as  procurement  funding. 
Appendix  D  provides  a  brief  discussion  of  the  association  of  post-MS  B  funding  climate 
and  RDT&E  cost  growth. 

Each  of  the  programs  for  which  the  database  includes  a  PAUC  growth  estimate 
completed  Engineering  and  Manufacturing  Development  (EMD),  went  into  production, 
and  fielded  at  least  some  units  to  operating  forces.  Programs  that  were  cancelled  (or 
programs  that  never  proceeded  beyond  EMD  or  which  were  not  a  Major  Defense 
Acquisition  Program  (MDAP))  were  not  used  in  the  statistical  analyses.  One  reason  to 
not  include  cancelled  programs  is  that  it  is  generally  very  difficult,  and  sometimes 
impossible,  to  get  a  reasonable  estimate  of  the  unit  cost  growth  they  experienced.  A  better 
reason  is  that  the  relevant  outcome  metric  for  these  programs  is  “cancellation.”  Another 
paper  in  this  series  (P-8280,  forthcoming)  will  examine  the  extent  to  which  changes  in 
acquisition  policy  and  process  are  associated  with  differences  in  cancellation  rates. 

The  database  used  in  this  research  contains  an  estimate  of  PAUC  growth  for  192 
MDAPs  that  entered  EMD  during  FY  1965-FY  2010.  We  follow  the  convention  of  not 
including  in  the  database  any  MDAP  that  was  not  at  least  five  years  beyond  EMD,  so  that 
cost  growth  would  have  time  to  appear.  The  most  recent  Selected  Acquisition  Reports 
(SARs)  available  for  this  paper  were  those  for  December  2015;  consequently,  MDAPs 
that  passed  MS  B  during  FY  2010  were  the  most  recent  included  in  the  database. 
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The  estimates  mainly  are  drawn  from  the  database  developed  for  IDA  Paper  P-5330 
(Revised),  which  in  turn  evolved  from  the  database  for  IDA  Paper  P-5126.1  The  cost 
growth  observations  for  FY  1965-FY  1969,  however,  and  a  few  of  the  observations  for 
FY  1965-FY  1989,  are  drawn  from  other  studies,  as  is  discussed  below. 

Figure  B-l  is  a  histogram  of  the  cost  growth  estimates  in  the  database  used  for  this 
paper.  The  paper  used  the  152  completed  MDAPs  that  passed  MS  B  during  the  period 
FY  1965-FY  2009  for  which  both  an  Average  Procurement  Unit  Cost  (APUC)  and  a 
PAUC  estimate  were  available. 


Figure  B-1.  Histogram  of  Quantity  Normalized  PAUC  Growth  from  the  MS  B  Baseline 


B.  Business  Rules 

Almost  all  of  the  data  used  in  this  research  were  taken  directly  or  indirectly  from 
SARs.  SARs  filed  in  FY  1997  and  subsequent  years  are  available  through  the  Defense 
Acquisition  Management  Information  Retrieval  (DAMIR)  system.  Many  SARs  filed 
before  FY  1997  are  available  on  an  Office  of  the  Under  Secretary  of  Defense 


1  David  L.  McNicol  et  at.,  “Further  Evidence  on  the  Effect  of  Acquisition  Policy  on  Cost  Growth  of 
Major  Defense  Acquisition  Programs,”  IDA  Paper  P-5330  (Revised)  (Alexandria,  VA:  Institute  for 
Defense  Analyses,  August  2016);  David  L.  McNicol  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD 
Acquisition  Policy  and  Process  on  Cost  Growth  of  Major  Defense  Acquisition  Programs,”  IDA  Paper 
P-5126  (Alexandria,  VA:  Institute  for  Defense  Analyses,  September  2014). 
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(Acquisition,  Technology  and  Logistics)  (OUSD(AT&L))  SIPRNet  site.  These  two 
sources  provided  SARs  under  about  345  distinct  labels. 

Not  all  of  these  distinct  labels  are  distinct  programs.  Three  steps  are  needed  to  get 
from  the  list  of  distinct  SAR  labels  to  a  list  of  MDAPs: 

1.  During  the  1970s,  each  Component  involved  in  a  joint  program  sometimes  filed 
a  SAR.  These  SARs  reported  the  same  program  data.  The  database  used  in  this 
research  includes  only  the  data  reported  (for  the  entire  program)  in  the  SAR 
filed  by  the  lead  Component. 

2.  The  program  name  used  on  the  SAR  often  changes  over  the  acquisition  cycle  for 
a  given  program.  For  example,  the  OH-58D  Kiowa  Warrior  was  first  reported  as 
the  Army  Helicopter  Improvement  Program  (AHIP).  In  most  cases  the  database 
uses  the  name  under  which  the  last  (or,  for  ongoing  programs,  most  recent)  SAR 
was  filed. 

3.  Multiple  MDAPs  that  have  passed  MS  B  are  sometimes  combined  into  a  single 
MDAP.  Conversely,  a  single  MDAP  that  has  passed  MS  B  is  sometimes  split 
into  two  or  more  separate  MDAPs.  If  the  data  permitted  (and  they  often  did  not), 
our  rule  was  to  maintain  the  program(s)  as  they  had  been  defined  at  MS  B. 

For  the  reasons  noted  above,  the  database  does  not  include  any  MDAPs  that  passed 
MS  B  after  FY  2010.  In  addition,  the  following  were  excluded  from  the  main  database: 

•  Major  Automated  Infonnation  Systems  (MAIS), 

•  Chemical  Demilitarization  Programs, 

•  Ballistic  Missile  Defense  programs  managed  by  the  Ballistic  Missile  Defense 
Agency  and  its  predecessors, 

•  Programs  that  filed  a  SAR  but  were  never  designated  as  an  MDAP,  and 

•  Programs  cancelled  before  they  passed  MS  B  or  before  they  were  designated  as 
an  MDAP. 

These  exclusions  were  indicated  by  the  purpose  of  the  analysis,  which  is  to  gauge  the 
effect  of  different  OSD-level  acquisition  regimes  and  funding  climates  on  MDAP 
outcomes.  The  database  then  should  include  only  programs  subject  to  OSD-level 
acquisition  policy  and  process.  To  at  least  a  significant  extent,  the  excluded  programs 
differed  from  the  MDAP  norm.  The  exclusions  therefore  resulted  in  a  main  database  that 
includes  316  MDAPs  that  entered  development  during  the  period  FY  1965-FY  2010. 

Most  of  the  MDAPs  in  the  database  passed  MS  B  at  the  OSD  level.  Some,  however, 
entered  at  MS  III/C,  obtained  both  MS  B  and  Low  Rate  Initial  Production  authority  in  a 
single  OSD-level  review,  or  passed  MS  B  at  the  Service  level  and  later  became 
Acquisition  Category  (ACAT)  I  programs.  These  cases  are  noted  in  the  database  for 
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programs  that  became  MDAPs  in  FY  1989  or  later,  but  not  reliably  noted  for  programs 
begun  earlier. 

Finally,  it  proved  to  be  necessary  to  adopt  a  clear  criterion  for  program  cancellation. 
In  the  database,  a  program  is  classified  as  cancelled  if: 

•  The  program  did  not  result  in  production  of  any  fully  configured  end  items,  or 

•  Any  fully  configured  end  items  produced  were  used  only  for  testing  and 
development. 

Application  of  this  definition  was  not  clear-cut  for  six  programs  that  passed  MS  B  at  the 
Service  level,  later  filed  SARs,  and  subsequently  were  cancelled.  We  retained  on  the  list 
of  cancelled  programs  the  five  that  had  been  designated  as  an  ACAT  I  program  and 
excluded  the  one  that  had  not.2 

Two  other  programs  were  counted  as  cancelled,  although  they  did  not  exactly 
satisfy  the  criteria  stated.  The  C-27J  was  included  on  the  list  of  cancelled  programs 
because  the  21  C-27Js  produced  were  placed  directly  in  long-tenn  storage  and  later 
transferred  to  Special  Operations  Command  and  the  US  Coast  Guard.  Roland  was 
included,  although  the  system  was  produced  in  the  United  States  in  limited  quantities  and 
issued  to  a  single  National  Guard  battalion,  which  falls  into  a  gray  area  between  issue  of 
the  system  to  active  duty  units  and  its  use  only  for  development,  experiment,  and 
training. 

The  file  “Program  Notes”  (provided  on  the  included  CD)  provides  some  information 
bearing  on  each  of  the  58  cancellations  we  identified.  We  found  16  additional  programs 
that  filed  one  or  more  SARs  during  FY  1959-FY  2009  and  were  cancelled.  These  16 
were  not  included  on  the  list  of  cancelled  programs  because  they  were  either  cancelled 
before  passing  MS  B,  were  never  designated  an  ACAT  I  program,  or  were  cancelled  after 
they  fell  below  the  ACAT  I  level.  They  appear  as  numbers  59-74  in  “Program  Notes.” 

C.  Coverage 

As  was  noted  above,  the  database  includes  58  MDAPs  that  were  cancelled  (as  an 
ACAT  I  program)  after  passing  MS  B  and  includes  258  programs  that  went  into 
production.  We  have  APUC  and  PAUC  estimates  for  184  of  the  MDAPs  that  went  into 
production,  of  which  32  were  still  underway  as  of  the  December  2015  SARs.  Table  B-l 
reports  the  relevant  data  broken  down  by  the  nine  time  periods  used  in  the  statistical 


2  AN/WQR-Advanced  Deployable  System,  AQM-127A  Supersonic  Low  Altitude  Target,  Advanced  Seal 
Delivery  System,  ASM-135A  Air-Launched  Anti-Satellite  System,  and  Land  Warrior  were  retained  on 
the  list  of  cancelled  programs.  Extended  Range  Munition  was  cancelled  before  it  was  designated  an 
ACAT  I  program. 
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analysis.  Overall,  the  database  reports  a  cost  growth  estimate  for  about  70  percent  of  the 
MDAPs  that  went  into  production. 


Table  B-1.  MDAPs  in  the  Database  Not  Cancelled,  with  an  APUC  and  a  PAUC  Estimate,  by 

Bust/Boom  Time  Periods 


Period  (FY) 

Went  into 
Production 

No.  with 
APUC  & 
PAUC 

Percent 
with  APUC 
&  PAUC 

1965-1969 

21 

16 

76% 

1970-1980 

63 

48 

76% 

1987-1989 

16 

10 

63% 

1990-1993 

14 

11 

78% 

1994-2000 

40 

30 

75% 

2001-2002 

11 

6 

55% 

2010 

3 

4 

75% 

Total 

168 

125 

74% 

1981-1982 

14 

7 

50% 

1983-1986 

45 

31 

69% 

2003-2009 

31 

21 

68% 

Total 

90 

59 

66% 

Grand  Total 

258 

184 

71% 

D.  Sources  of  Cost  Growth  Estimates 

Table  B-2  reports  the  sources  of  the  APUC  and  PAUC  estimates  used  in  this  paper. 
Nearly  half  of  the  total  was  taken  from  an  MDAP  cost  growth  database  developed  and 
maintained  by  the  Office  of  Program  Analysis  and  Evaluation  (PA&E)  Resource 
Analysis  deputate.  The  PA&E  cost  growth  database  is  documented  in  a  briefing  by  John 
McCrillis  given  at  the  2003  Annual  DoD  Cost  Analysis  Symposium.3  The  briefing  is 
included  on  the  CD  provided  on  the  inside  back  cover  of  this  paper. 


3 

John  McCrillis,  “Cost  Growth  of  Major  Defense  Programs,”  Briefing  (presented  at  the  Department  of 
Defense  Cost  Analysis  Symposium,  Williamsburg,  VA,  January  30,  2003). 
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Table  B-2.  Sources  of  the  APUC  and  PAUC  Growth  Estimates  Used  in  Different  Periods 


Period  (FY) 

PA&E 

P-2722 

RAND 

In-House 

Total 

1964-1969 

0 

16 

0 

0 

16 

1970-1979 

36 

8 

2 

0 

46 

1980-1989 

45 

0 

4 

1 

51 

1990-1999 

7 

0 

0 

32 

36 

2000-2009 

0 

0 

0 

30 

25 

2010 

0 

0 

0 

3 

3 

Total 

88 

24 

6 

66 

184 

APUC  and  PAUC  growth  estimates  for  an  additional  24  MDAPs  were  taken  from 
IDA  Paper  P-2722.4  The  provided  CD  includes  the  main  volume  of  P-2722,  as  well  as  an 
Excel  workbook  with  the  data.  The  next  section  of  this  appendix  describes  how  the 
P-5126  cost  growth  estimates  were  made. 

Communication  from  the  RAND  Corporation  provided  updates  to  the  FY  2015 
SARs  of  estimates  for  six  MDAPs  published  in  a  1996  study5  of  APUC  and  PAUC 
growth  estimates  normalized  to  the  MS  B  baseline. 

Finally,  APUC  and  PAUC  growth  estimates  for  66  MDAPs  were  made  in-house  as 
part  of  the  work  on  this  project.  Fifty-eight  of  the  MDAPs  in  the  PA&E  cost  growth 
database  were  still  ongoing  at  the  time  of  the  final  PA&E  update  (that  is,  when  the 
December  2004  SARs  were  filed).  These  were  updated  with  data  from  the  December 
2015  SARs.  In  addition,  APUC  and  PAUC  growth  estimates  for  MDAPs  that  passed  MS 
B  during  FY  2008-FY  2010  were  made,  again  using  the  December  2015  SARs. 

The  PA&E  estimates  were  constructed  through  a  detailed  examination  of  the  SAR 
variances.  The  IDA  P-2722,  IDA  P-5126,  and  RAND  estimates  were  made  with  data  at  a 
much  more  aggregated  level.  The  methods  used  were  essentially  the  same,  but  it  is 
reasonable  to  assume  that  they  differ  in  detailed  ways  not  captured  by  the  general 
characterization  each  offers  of  the  method  used.  P-2722  did  not  in  all  cases  follow  the 
business  rules  used  in  P-5 126  and  this  paper. 

These  four  sources  use  the  same  definitions  of  the  relevant  cost  tenns  and  are  based 
on  SAR  data.  Each  also,  in  most  instances,  measures  cost  growth  from  the  MS  B  baseline 
when  it  is  available  and  reports  quantity  nonnalized  unit  cost  growth.  Thus,  a  PAUC 


4  Karen  Tyson  et  al.,  “The  Effects  of  Management  Initiatives  on  the  Costs  and  Schedules  of  Defense 
Acquisition  Programs,  Vol.  I:  Main  Report,”  IDA  Paper  P-2722  (Alexandria,  VA:  Institute  for  Defense 
Analyses,  1992). 

5  Jeanne  M.  Jarvaise,  Jeffrey  A.  Drezner,  and  Dan  Norton,  “The  Defense  System  Cost  Performance 
Database:  Cost  Growth  Using  Selected  Acquisition  Reports,”  RAND  Report  MR-625-OSD  (Santa 
Monica,  CA:  The  RAND  Corporation,  1996). 
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estimate  from,  for  example,  P-2722  means  the  same  thing  as  an  APUC  estimate  from  the 
other  three  sources. 

There  were  several  MDAPs  from  the  1960s  and  1970s  for  which  we  had  two  APUC 
and  PAUC  growth  estimates.  The  decisions  on  which  of  the  alternative  estimates  to  use 
was  entirely  rules-based.  The  PA&E  database  did  not  provide  estimates  for  MDAPs  that 
entered  EMD  during  FY  1965-FY  1969.  The  unit  cost  growth  estimates  used  for 
FY  1965-FY  1969  are  from  P-2722.  In  addition  to  the  SAR  data,  P-2722’s  estimates  in 
many  cases  reflected  other  sources  of  infonnation,  including  material  provided  by  the 
program  office  and  contractors.  For  FY  1970  and  beyond,  we  used  the  PA&E  estimate  in 
all  cases  in  which  the  last  SAR  for  the  program  had  been  filed  by  the  time  of  the  final 
update  of  the  PA&E  database  (which  used  the  December  2004  SARs).  In  a  few  cases, 
P-2722  had  a  cost  growth  estimate  for  a  program  not  included  in  the  PA&E  database.  In 
these  instances,  we  used  the  estimate  from  P-2722  if  the  program  was  reported  complete 
in  the  most  recent  SARs  used  in  making  the  estimate;  otherwise,  we  used  the  RAND 
estimate  if  available. 

E.  Computation  of  the  P-5126  (Main  Database)  V  5.2  Estimates 

This  section  briefly  describes  how  the  66  in-house  estimates  were  made.  The 
relevant  data  and  computations  are  in  the  “Data  and  Computations  for  In-house  Cost 
Growth  Estimates. xlsx”  file  on  the  CD. 

1.  RDT&E 

The  SARs  report  fully  configured  units  acquired  with  RDT&E  funds  and  those 
acquired  with  procurement  funds.  Only  the  former  are  used  in  computing  quantity- 
adjusted  RDT&E  cost  growth.  Our  procedure  was  simply  to  compute  the  ratio  of  the 
Current  Estimate  (CE)  of  RDT&E  cost  and  the  baseline  RDT&E  cost  (both  in  program 
base  year  dollars)  and  scale  that  by  the  ratio  of  baseline  quantity  to  CE  quantity.  Suppose, 
for  example,  that  the  number  of  fully  configured  units  purchased  with  RDT&E  funds  has 
increased  from  four  to  five  and  that  CE  RDT&E  cost  is  50  percent  larger  than  the 
baseline  cost.  Our  computation  of  unit  RDT&E  cost  growth  is  then  (4/5)  x  1.5  -  1,  or  20 
percent. 

2.  APUC 

The  method  used  to  nonnalize  APUC  for  quantity  change  depended,  first,  on  the 
extent  to  which  quantity  changed  between  MS  B  and  the  final  SAR  and,  second,  on 
whether  a  useable  estimate  of  the  slope  of  the  learning  curve  was  available. 
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a.  No  Quantity  Change  (NQC) 

The  SAR  CE  quantity  was  within  ±1  percent  of  the  MS  B  quantity  for  12  of  the 
MDAPs  for  which  estimates  were  required.  No  quantity  normalization  is  needed  for  these 
programs;  their  APUC  growth  is  computed  by  dividing  the  CE  APUC  in  the  final  SAR 
(or  the  December  2012  SAR  for  an  ongoing  program)  by  the  MS  B  APUC  and 
subtracting  1.  The  APUC  growth  for  SBIRS-High  also  falls  under  this  heading.  The  total 
number  of  SBIRS-High  satellites  to  be  acquired  decreased  from  five  (at  MS  B)  to  four 
(the  December  2012  SAR).  The  decrease,  however,  was  in  a  satellite  purchased  with 
RDT&E  funds,  and  we  did  not  put  these  on  a  learning  curve.  There  was  no  change  in  the 
number  of  SBIRS-High  satellites  purchased  with  procurement  funds.  Finally,  although 
the  PAC-3  quantity  change  fell  outside  the  ±1  percent  boundary,  data  limitations  made  it 
necessary  to  compute  the  PAC-3  APUC  growth  as  the  ratio  of  the  CE  and  MS  II  APUCs. 

b.  Defense  Acquisition  Management  Information  Retrieval  System  (DAMIRS) 
Learning  Curve  (DLC) 

The  DoD  contractor  staff  for  DAMIRS  provided  us  with  their  estimates  of  learning 
curve  parameters  that  we  were  able  to  use  to  compute  APUC  growth  for  13  MDAPs  that 
passed  MS  B  during  FY  1989-FY  2001.  We  refer  to  these  as  the  DAMIRS  Learning 
Curve  (DLC)  APUC  growth  estimates.  For  each  of  these,  we  took  the  CE  APUC  growth 
in  program  base  year  dollars  from  the  last  SAR  for  the  program  or  the  December  2015 
SAR  (for  ongoing  programs).  The  task  was  to  nonnalize  this  APUC  estimate  to  the  MS  B 
quantity,  which  was  done  as  follows: 

•  We  used  the  learning  curve  to  compute  the  recurring  flyaway  cost  at  the  MS  B 
baseline  quantity. 

•  The  CE  estimates  of  RDT&E  and  non-recurring  flyaway  cost  were  taken  from 
the  final  SAR  for  the  program  or  from  the  December  2012  SAR  (for  ongoing 
programs). 

•  Support  costs  paid  for  with  procurement  dollars  are,  for  many  programs, 
primarily  initial  spares  and  support  equipment,  although  other  items  may  also 
fall  into  this  category.  Initial  spares  and  support  equipment  normally  scale  with 
the  number  of  units  of  the  system  purchased.  For  that  reason,  we  used  the  CE 
support  cost  reported  in  the  last  or  most  recent  SAR  scaled  to  the  MS  B  baseline 
quantity. 

c.  Calibrated  Learning  Curve  (CLC) 

Twenty-nine  MDAPs  did  not  have  a  PA&E  estimate  or  estimated  learning  curve 
parameters,  and  their  CE  quantity  was  significantly  different  from  the  MS  B  quantity. 
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The  approach  we  used  in  those  cases  rested  on  a  cost  progress  curve  of  the  conventional 
form: 


C  =  TQP.  (B-l) 

In  this  expression,  C  is  recurring  flyaway  cost,  T  is  first  unit  cost,  Q  is  cumulative 
production,  and  P  is  the  cost  progress  parameter.  We  solved  this  and  used  the  CE  for 
recurring  flyaway  to  get: 

f  =  CQ~P.  (B-2) 

This  will  be  referred  to  as  the  calibrated  learning  curve  (CLC)  method.  A  value  of  P  = 
0.94  was  used  for  each  of  the  programs.  From  this  point,  the  computations  were  the  same 
as  those  for  MDAPs  for  which  DAMIRS  staff  provided  the  learning  curve  parameters. 

3.  PAUC 

Quantity  PAUC  is  simply  the  sum  of  quantity  nonnalized  RDT&E  and 
Procurement,  divided  by  the  baseline  quantity,  less  1 . 

4.  Summary 

Table  B-3  provides  an  overview  of  the  number  of  estimates  in  P-5126  made  with 
each  of  the  methods. 


Table  B-3.  Sources  of  the  Quantity  Normalized  Unit  Cost  Growth  Estimates  Used  in 

Different  Periods 


Period  (FY) 

NQC 

DLC 

CLC 

Total 

1989-2001 

5 

13 

21 

39 

2002-2010 

7 

0 

20 

27 

Total 

12 

13 

41 

66 

F.  Comparison  of  the  PA&E  and  CLC/DLC  PAUC  Growth  Estimates 

P-5126  compared  the  PA&E  estimates  for  23  MDAPs  with  estimates  made  using 
the  CLC  and  DLC  methods.6  That  material  is  repeated  here  without  substantial  changes. 

The  obvious  approach  is  to  compare  the  PA&E  PAUC  growth  for  systems  that  have 
been  completed  with  PAUC  growth  for  those  same  systems  computed  using  the  DLC  and 
CLC  methods.  Unfortunately,  there  are  no  MDAPs  that  have  been  completed  and  for 


6  McNicol  and  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition  Policy  and  Process,”  A-7,  A-9. 
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which  we  have  both  a  PA&E  PAUC  growth  estimate  and  the  data  needed  to  compute  a 
DLC  or  CLC  estimate.  The  best  we  can  do  is  to  examine  the  23  MDAPs  that  passed  MS 
II/B  during  FY  1989-FY  2001  and  for  which  we  have  a  PA&E  PAUC  growth  estimate,  a 
DLC  estimate,  and  a  CLC  estimate. 

The  PA&E  estimates  were  most  recently  updated  with  the  2004  SARs.  The  DLC 
and  CLC  estimates,  in  contrast,  incorporated  more  recent  data — either  the  final  SAR  for 
the  program  or,  for  ongoing  programs,  the  December  2012  SAR.  Consequently,  in  most 
cases  we  would  expect  the  DLC  and  CLC  PAUC  growth  estimates  to  be  larger  than  the 
corresponding  PA&E  estimate.  That  is  the  test:  A  method  fails  if  it  yields  estimates  that 
are  “too  often”  and  by  “too  much”  less  than  the  PA&E  estimates.  Clearly,  this  is  a  weak 
test. 

The  relevant  estimates  are  presented  in  Table  B-4.  The  comparison  of  the  PA&E 
estimates  and  CLC  estimates  is  on  the  left,  and  the  comparison  of  the  PA&E  and  DLC 
estimates  is  on  the  right.  The  CLC  estimates  are  larger  than  the  PA&E  estimates  for  17  of 
the  23  MDAPs — in  most  cases,  considerably  larger.  They  are  smaller  in  six  cases  (shaded 
rows).  In  all  but  one  of  these  cases  (Joint  Direct  Attack  Munition,  or  JDAM)  the 
differences  are  absolutely  or  relatively  small.  The  average  of  CLC  PAUC  growth 
estimates  is  77  percent,  in  comparison  to  an  average  of  60  percent  for  the  PA&E 
estimates.  The  DLC  estimates  exhibit  the  same  pattern.  The  average  of  the  DLC 
estimates  is  73  percent,  and  four  of  them  (shaded  rows)  are  less  than  the  PA&E  estimate 
for  the  program,  three  by  a  substantial  amount. 
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Table  B-4.  Comparison  of  PA&E,  CLC,  and  DLC  PAUC  Growth  Estimates  for  23  MDAPs 


Program 

PA&E 

CLC 

Program 

PA&E 

DLC 

Longbow  Apache 

78% 

117% 

Longbow  Apache 

78% 

133% 

F-22 

41% 

71% 

F-22 

41% 

55% 

F/A-18E/F 

6% 

12% 

F/A-18E/F 

6% 

9% 

Bradley  Upgrade 

39% 

54% 

Bradley  Upgrade 

39% 

86% 

MIDS 

30% 

72% 

MIDS 

30% 

68% 

CEC 

48% 

62% 

CEC 

48% 

62% 

H-1  Upgrades 

124% 

192% 

H-1  Upgrades 

124% 

197% 

LPD  17 

43% 

71% 

LPD  17 

43% 

72% 

CH-47F 

147% 

173% 

CH-47F 

147% 

156% 

GMLRS/GMLRS  AW 

125% 

249% 

GMLRS/GMLRS  AW 

125% 

243% 

MH-60S 

62% 

69% 

MH-60S 

62% 

70% 

Tactical  Tomahawk 

24% 

28% 

Tactical  Tomahawk 

24% 

27% 

GBS 

10% 

31% 

GBS 

10% 

33% 

Stryker 

21% 

25% 

Stryker 

21% 

22% 

UH-60M  Black  Hawk 

49% 

62% 

UH-60M  Black  Hawk 

49% 

61% 

WGS 

28% 

55% 

WGS 

28% 

42% 

C-130J 

70% 

84% 

C-130J 

70% 

70% 

JPATS 

43% 

40% 

JPATS 

43% 

44% 

SSN  774 

35% 

33% 

SSN  774 

35% 

37% 

JDAM 

18% 

-10% 

JDAM 

18% 

-13% 

Javelin 

229% 

197% 

Javelin 

229% 

134% 

MH-60R 

95% 

74% 

MH-60R 

95% 

80% 

NAS 

25% 

21% 

NAS 

25% 

1% 

Average 

60% 

77% 

60% 

73% 

Note:  The  PA&E  estimates  were  updated  only  through  the  2004  SARs.  The  CLC  and  DLC  estimates 
incorporate  information  from  the  last  SARforthe  program  or  the  December  2012  SAR  (for  ongoing 
programs). 
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Appendix  C. 

Boom  Effects  for  McNamara-Clifford, 
Acquisition  Reform  (AR),  and  the  Bust  Phase  of 

the  DAB  Post  AR 


Table  C-l  presents  average  Program  Acquisition  Unit  Cost  (PAUC)  growth  and 
average  program  duration  data  for  the  McNamara-Clifford  and  the  Acquisition  Reform 
(AR)  periods. 

Table  C-1.  Average  PAUC  Growth  and  Program  Duration  for  Completed  Programs  for 

McNamara-Clifford  and  AR 

Average  PAUC 

Growth  Average  Duration 


McNamara-Clifford 

BustO 

87%  (12) 

9.3 

Bustl 

34%  (4) 

20.5 

Acquisition  Reform 

BustO 

2%  (1) 

7 

Bustl 

38%  (18) 

14.6 

*  Quantity  APUC  from  the  MS  B  baseline. 

**  From  MS  B  through  the  year  in  which  the  program’s  last  SAR  was  filed. 


In  contrast  to  what  was  found  for  the  DS ARC/DAB -Bust  period,  for  McNamara- 
Clifford,  average  PAUC  growth  for  BustO  programs  is  about  two  and  one-half  times  that 
of  Bustl  programs.  The  difference  is  statistically  significant. 1  This  may  be  due  to  the  fact 
that  the  Bustl  programs  continued  into  at  least  the  early  1980s  and  therefore  presumably 
were  more  strongly  influenced  by  the  1969  Packard  acquisition  reforms,  which  are 
associated  with  a  significant  reduction  in  PAUC  growth. 

The  cost  growth  data  for  AR  are  not  useful  for  statistical  analysis  because  only  one 
program  that  passed  MS  B  during  that  period  (AV-8B  Remanufacture)  had  been 
completed  by  the  December  2015  SARs. 


1  K-S  and  A-D  find  the  distributions  of  PAUC  growth  in  BustO  and  Bust  1,  respectively,  to  be  consistent 
with  a  normal  distribution.  P  =  0.048  for  a  two-tailed  t-test  with  correction  for  unequal  variances. 
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Appendix  D. 

RDT&E  Cost  Growth  for  the  DS ARC/DAB 

Period 


Table  D-l  presents  data  on  Research,  Development,  Test  and  Evaluation  (RDT&E) 
cost  growth  and  duration  in  the  DSARC/DAB  period  that  parallel  the  PAUC  and  duration 
data  presented  in  Table  2  (on  page  5),  Table  3  (on  page  6),  and  Table  4  (on  page  7).  The 
number  of  observations  in  some  cells  differs  from  that  given  for  PAUC  because  the 
database  does  not  have  an  RDT&E  estimate  for  all  programs  for  which  there  is  a  PAUC 
growth  estimate. 

Table  D-1.  Average  RDT&E  Growth  and  Average  Program  Duration  by  Number  of  Boom 
Periods  Encountered  for  Bust  and  Boom  Climates 


Bin 

Average  RDT&E 
Growth3 

Average  Duration13 

1st  Bust  Period 

BustO 

5%  (5) 

6.7 

Bustl 

53%  (38) 

14.1 

2nd  Bust  Period 

BustO 

37%  (7) 

8.0 

Bustl 

45%  (17) 

15.8 

1st  Boom  Period 

BoomO 

41%  (26) 

9.6 

Booml 

65%  (7) 

22.3 

2nd  Boom  Period 

BoomO 

1%  (9) 

66.1 

Booml 

n/a  (0) 

n/a 

a  Quantity  APUC  from  the  MS  B  baseline. 

b  From  MS  B  through  the  year  in  which  the  program’s  last  SAR  was  filed. 


The  pattern  of  growth  in  RDT&E  in  the  first  bust  period  is  consistent  with  that 
observed  for  PAUC  growth:  (1)  average  RDT&E  growth  for  programs  in  Bustl  is 
significantly  higher  than  the  average  for  BustO;  and  (2)  the  proportion  of  programs  of 
Bustl  that  fall  into  the  right  tail  of  the  distribution  also  is  significantly  higher  than  it  is  for 
BustO. 1 


M-W  UP  =  0.025  (U  =  35.5,  m  =  38,  m  =  5).  P  =  0.051  for  Fisher’s  Exact  Test  (FET)  using  the  number 
of  programs  in  BustO  and  Bustl  with  an  RDT&E  growth  of  at  least  40  percent. 
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Average  RDT&E  growth  in  the  second  bust  period  is  noticeably  higher  in  Bustl 
than  in  BustO,  but  the  difference  is  not  statistically  significant.  The  proportion  of 
programs  with  RDT&E  cost  growth  of  more  than  40  percent  also  is  not  significantly 
higher  in  Bustl  than  in  BustO.2 

In  the  first  boom  period,  average  RDT&E  cost  growth  is  significantly  higher  for 
MDAPs  in  Booml  than  for  those  in  BoomO,  and  the  proportion  of  MDAPs  with  RDT&E 
growth  of  at  least  40  percent  also  is  significantly  higher  in  Booml  than  in  BoomO.3 


2  M-W  UP  =  0.308  (U  =  43,  m  =  17, 112  =  7).  P  =  1.000  for  FET  using  the  number  of  programs  in  BustO 
and  Bustl  with  an  RDT&E  growth  of  at  least  40  percent. 

3  M-W  UP  =  0.075  (U  =  132,  m  =  26,  n?  =  7).  P  =  0.027  for  FET  using  the  number  ofprograms  in  BustO 
and  Bustl  with  an  RDT&E  growth  of  at  least  40  percent. 
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ACAT 

Acquisition  Category 

A-D 

Anderson-Darling  Test 

AHIP 

Army  Helicopter  Improvement  Program 

ANOVA 

Analysis  of  Variance 

APUC 

Average  Procurement  Unit  Cost 

AR 

Acquisition  Reform 

ATM 

Asynchronous  Transfer  Mode 

BA 
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CPIF 

Cost  Plus  Incentive  Fee 
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Defense  Acquisition  Management  Infonnation  Retrieval 
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DDR&E 

Director,  Defense  Research  and  Engineering 
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DoD 

Department  of  Defense 

DoDD 

Department  of  Defense  Directive 
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Department  of  Defense  Instruction 
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FFP 

Finn  Fixed  Price 

FPIF 

Fixed  Price  Incentive  Fee 
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MAIS 

Major  Automated  Information  System 
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MDAP 

Major  Defense  Acquisition  Program 

MILPERS 
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MS 
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OSA 

Office  of  Systems  Analysis 

OSD 

Office  of  the  Secretary  of  Defense 
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Program  Analysis  and  Evaluation 

PAUC 

Program  Acquisition  Unit  Cost 
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Logistics) 

VCJCS 
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